Tempelis

@Tempelis has joined the channel

vrabbi

2020-11-04 16:36:01

@vrabbi has joined the channel

jsturtevant

2020-11-04 16:37:28

@jsturtevant has joined the channel

neolit123

2020-11-04 16:40:52

@neolit123 has joined the channel

neolit123

2020-11-04 16:42:05

set the channel topic: Slack channel for the image-builder project: https://github.com/kubernetes-sigs/image-builder

cecile

2020-11-04 16:43:10

@cecile has joined the channel

dongsupark

2020-11-04 16:43:19

@dongsupark has joined the channel

vbatts

2020-11-04 16:43:34

@vbatts has joined the channel

vbatts

2020-11-04 16:43:38

jsturtevant

2020-11-04 16:52:15

I'll kick it off from the thread I started in sig-cluster-lifecycle: could I get a review/approval for the image builder for windows ? If there is anything holding it back please let me know.
Happy to answer questions or address any feedback. Thanks!

jsturtevant (https://github.com/jsturtevant)

#382 Windows image configuration scripts and Azure builder

Initial support for configuring Windows nodes for CAPI. This is the first step for Windows support as described in Windows Proposal in CAPI: <a href="https://github.com/kubernetes-sigs/cluster-api/pull/3616">kubernetes-sigs/cluster-api#3616</a> and partly solves <a href="https://github.com/kubernetes-sigs/image-builder/issues/30">#30</a>. ~The current scripts work to create an image but I am still testing that the image works with a provider like CAPZ. I wanted to open the PR to get feedback on the setup.~ Update: now tested via CAPZ <a href="https://github.com/kubernetes-sigs/image-builder/pull/382#issuecomment-707245381">#382 (comment)</a> A few notes: • <a href="https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/">Kubeadm support for Windows</a> is currently uses <a href="https://github.com/kubernetes-sigs/sig-windows-tools/blob/master/kubeadm/scripts/PrepareNode.ps1">https://github.com/kubernetes-sigs/sig-windows-tools/blob/master/kubeadm/scripts/PrepareNode.ps1</a>. I referred to that script for some of the set up. • Most windows users are use to scripts for configuration over ansible. I initially started this with scripts but the customization required in the scripts and use with Packer was quickly making it unmaintainable. The final ansible solutions feels much cleaner to me and closer to the Linux configuration which makes it easier to grok. • The Ansible and packer configuration has been split out since there is not enough in common. I think having them in separate folder will make the maintenance easier longer. Initially tried to keep the packer configuration re-use high but ended splitting it out into it's own folder under <code>config/windows</code> to keep it cleaner since there were so many different setting between Windows/linux. • The ansible files for windows are in an awkward spot, It would look cleaner if there was a clear split between Linux and Windows under the <code>ansible</code> folder. Didn't include that change here to minimize the diff. Looking forward to feedback.

Assignees

codenrhoden

Labels

cncf-cla: yes, size/XXL

cecile

2020-11-04 17:29:09

lgtm but didn’t approve to get more eyes on it

cecile

2020-11-04 17:30:18

I’m hoping at least @codenrhoden and/or @moshloop / @jdetiber reviews it, and maybe someone who is more familiar with the windows specific scripts than I am like @Kalya Subramanian or @Mark Rossetti

jdetiber

2020-11-04 17:31:57

Will take another look at it this afternoon

moshloop

2020-11-04 17:34:04

It lgtm, but my windows and powershell is rudimentary at best

Kalya Subramanian

2020-11-04 17:34:31

Will take a look today

jsturtevant

2020-11-04 17:45:44

Thanks all!

Mark Rossetti

2020-11-04 23:10:52

I can take a look today

jdetiber

2020-11-04 17:19:51

@jdetiber has joined the channel

moshloop

2020-11-04 17:31:56

@moshloop has joined the channel

Kalya Subramanian

2020-11-04 17:34:23

@Kalya Subramanian has joined the channel

cecile

2020-11-04 21:32:34

heads up I’m seeing the Azure 20.04 build fail in a few PRs with [0;32m vhd-ubuntu-2004: fatal: [default]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unknown reason"}[0m

cecile

2020-11-04 21:33:12

The timing is strange but I don’t believe this is related to my recent job config update, it looks like a legitimate packer failure

cecile

2020-11-04 21:33:48

nothing has changed AFAIK in the image-builder scripts so this might be an issue with either a new apt package or with a new base image for ubuntu 20.04

cecile

2020-11-04 21:42:48

CecileRobertMichon (https://github.com/CecileRobertMichon)

#426 CAPI: temporarily disable Azure Ubuntu 20.04 build

The Azure 20.04 CI build is failing with <code>vhd-ubuntu-2004: fatal: [default]: FAILED! =&gt; {"changed": false, "msg": "Failed to update apt cache: unknown reason"}</code> This looks like it might be an issue with either a new apt package or with a new base image for ubuntu 20.04. Temporarily disable the test while we investigate to avoid blocking PRs.

Labels

cncf-cla: yes

Mark Rossetti

2020-11-04 23:10:44

@Mark Rossetti has joined the channel

codenrhoden

2020-11-05 15:12:06

@codenrhoden has joined the channel

codenrhoden

2020-11-05 15:15:22

yay new channel! @cecile @moshloop just wanted to say I am sorry I still haven’t been active in GitHub doing reviews/development. I’ve still had some ongoing health problems, and I even spent yesterday getting a CT scan and scheduling a follow-up surgery for next Tuesday. 😞 Hopefully after that (plus a bit of healing time) things will return to normal.

cecile

2020-11-05 16:57:56

No problem at all, please take care of yourself! Your health is top priority, everything else can wait. Good luck with the surgery, I hope everything comes back to normal soon for you 🤞

🤞 dongsupark, Maximilian Rink, fabio

:thanks: codenrhoden

kiran keshavamurthy

2020-11-05 15:16:17

@kiran keshavamurthy has joined the channel

Tushar Aggarwal

2020-11-05 15:16:17

@Tushar Aggarwal has joined the channel

Maximilian Rink

2020-11-05 18:45:14

@Maximilian Rink has joined the channel

voor

2020-11-09 17:24:41

@voor has joined the channel

naadir

2020-11-09 17:27:35

@naadir has joined the channel

shysank

2020-11-09 17:29:52

@shysank has joined the channel

Ratnopam

2020-11-10 03:08:01

@Ratnopam has joined the channel

hidekazuna

2020-11-10 06:50:47

@hidekazuna has joined the channel

fabio

2020-11-10 09:33:31

@fabio has joined the channel

nader ziada

2020-11-10 15:31:59

@nader ziada has joined the channel

Lauri Apple

2020-11-12 12:00:51

@Lauri Apple has joined the channel

Andy Townsend

2020-11-16 11:06:54

@Andy Townsend has joined the channel

codenrhoden

2020-11-16 19:30:46

Should we cancel this week’s office hours due to Kubecon NA?

👍 dongsupark

Tushar Aggarwal

2020-11-16 19:36:41

Yes 👍

dongsupark

2020-11-16 20:04:27

Gentle reminder: in the last bi-weekly call, we agreed to merge the Flatcar PR , and address remaining things afterwards incrementally. It would be great to see it merged. 🙂

dongsupark (https://github.com/dongsupark)

#248 capi: support Flatcar Container Linux

This is a cluster-API image builder for <a href="https://www.flatcar-linux.org">Flatcar Container Linux</a>. On the Packer side it includes <code>packer/qemu/qemu-flatcar.json</code> and <code>packer/qemu/packer.json</code>. In <code>packer/config/</code> it also includes some other JSON files with configuration for ansible and packages installed with Ansible. On the Ansible side there's some preliminary playbook in <code>ansible/</code>. notes • <code>images/capi/ansible/roles/setup/tasks/bootstrap-flatcar.yml</code> installs python under <code>/opt</code>. • <code>images/capi/ansible/roles/kubernetes/tasks/url.yml</code> installs artificts of Kubernetes and CNI under <code>/opt</code> and <code>/etc</code>, mainly because <code>/usr</code> is read-only in Flatcar. • It unpacks only parts of containerd binaries, since Flatcar already has its own built-in containerd. • To make containerd work correctly, we install Flatcar-specific config files for containerd, so it listens on its containerd socket <code>/run/docker/libcontainerd/docker-containerd.sock</code>. • Since Flatcar has its own containerd socket, we need to also specify the socket name when running commands like <code>kubeadm config images pull</code>, <code>/opt/bin/ctr images import</code> as well as in <code>/etc/crictl.yaml</code>. • We do not delete <code>/etc/kubeadm.yml</code> for Flatcar, because later steps running <code>kubeadm init</code> require the config to be in place. TODOs • Make sure it works for other platforms than qemu & AWS /cc <a href="https://github.com/vbatts">@vbatts</a> <a href="https://github.com/t-lo">@t-lo</a> <a href="https://github.com/dims">@dims</a> <a href="https://github.com/detiber">@detiber</a> <a href="https://github.com/codenrhoden">@codenrhoden</a>

Labels

cncf-cla: yes, size/XL

Comments

👀 codenrhoden

jsturtevant

2020-11-17 17:25:32

Ready with the Windows PR as well. Thanks to everyone that has reviewed it and tried out out so far!

jsturtevant (https://github.com/jsturtevant)

#382 Windows image configuration scripts and Azure builder

Assignees

codenrhoden

Labels

cncf-cla: yes, size/XXL

:ack: codenrhoden

Tushar Aggarwal

2020-11-19 18:02:07

Hi folks, I and @moshloop will be giving a talk on "Deep dive: K8s Image builder" at Kubecon'20 for those who are attending.
Time: 2:55 PM EST / 11:55 PDT

kccncna20.sched.com

KubeCon + CloudNativeCon North America 2020 Virtual: Image-Builder Deep Dive - Moshe Immerman...

View more about this event at KubeCon + CloudNativeCon North America 2020 Virtual

Original URL: https://kccncna20.sched.com/event/ekGy/image-builder-deep-dive-moshe-immerman-flanksource-tushar-aggarwal-vmware?iframe=no

👍 cecile, dongsupark, naadir, voor

hidekazuna

2020-11-20 00:27:56

Please approve the PR to fix to build ubuntu 20.04 CAPI image for OpenStack.
Simply updated to 20.04.1 and it's checksum since 20.04 ISO image checksum has gone.

prankul88 (https://github.com/prankul88)

#435 Update Ubuntu20 ISO image

This PR resolves the error which occurs while trying to build ubuntu20 image using <code>make build-qemu-ubuntu-2004</code>

Assignees

hidekazuna

Labels

cncf-cla: yes, lgtm, size/XS

David Lai

2020-11-20 23:49:22

@David Lai has joined the channel

Jura

2020-11-21 13:07:24

@Jura has joined the channel

Deepak Sharma

2020-11-23 14:14:13

@Deepak Sharma has joined the channel

Maximilian Rink

2020-11-30 14:42:35

We have found a few bugs with ubuntu and the current http proxy handeling and i've created an PR that fixes them 😄

MaxRink (https://github.com/MaxRink)

#440 Fix non-working Proxy-Flags in Ansible

While moving our image buildung into an environment that needs an HTTP Proxy we discovered that the provisions in the Ansible are not fully working. This PR fixes that

Labels

cncf-cla: no, needs-ok-to-test, size/S

Comments

Maximilian Rink

2020-12-02 10:47:18

In the quest to make our images build behind an http proxy ive stumbled across another annoyance:
Kubeadm image pull doesnt respect the proxy as its set right now.
Im unsure wht would be the best way to handle this. I guess the easiest way would just be to skip the pulling behind http proxies and then rely on having a working registry for actual bootstrap

voor

2020-12-02 12:24:00

Yes, you should be able to skip image pull with kubeadm, we use a custom role to pull images in a different manner

Maximilian Rink

2020-12-02 12:38:08

and goss also doent play nicely with a proxy used :S

Maximilian Rink

2020-12-02 12:46:16

hmm, its correctly set for my CI, it just seems like packer is not passing it to the GOSS provisioner. Has any of you encountered that behaviour yet?

naadir

2020-12-02 13:27:07

have you set the proxy on containerd ?

naadir

2020-12-02 13:27:37

kubeadm is doing crictl pull, so a proxy setting on kubeadm itself should be ineffective

Maximilian Rink

2020-12-02 13:36:57

no, but i have hoped to avoid that 😄
As i need to modify the systemd service for that

looks like i have to grow my PR once more 😄

naadir

2020-12-02 13:38:12

another way would be to pull the image locally, export it, copy to the machine, import it and then run kubeadm

Maximilian Rink

2020-12-02 13:46:51

yeah, that even more errorprone 😄

Maximilian Rink

2020-12-02 13:47:17

should also do

Maximilian Rink

2020-12-02 13:48:07

that only leaves me with goss to figure out 🤔

Maximilian Rink

2020-12-02 16:07:48

seems like it is not possible to setup the goss provisioner so that the inital download support an proxy
It has env flags, but those arent in play at that stage 😕

The best workaround that i can currently see it to fetch the goss binary in ansible and then set the provisioner to skip the download and to use the prefetched binary

naadir

2020-12-02 16:56:22

eek

Maximilian Rink

2020-12-02 17:02:47

http proxies are ugly tech 😄

Maximilian Rink

2020-12-02 17:02:55

But i dont have a choice

Maximilian Rink

2020-12-02 17:05:16

In the same process ive noticed that packer fetches the ovf directly from the node and not from the vcenter like a lot of other toling does. And it doesnt fall back on the vcenter pull. Have you noticed the same @naadir or have i just a weird packer config running?

naadir

2020-12-02 17:05:54

i was under the impression it functions as a 301 redirect unless you're using a content library?

Maximilian Rink

2020-12-02 17:06:47

pretty sure it proxies it. I can export ovf fine, altough i only have a connection to the vcenter and cant reach any of the hosts directly

naadir

2020-12-02 17:07:08

if you download from the vcenter ui, you'll also get an error unless you accept the esxi tls cert for the host vcenter is telling the browser to download from

naadir

2020-12-02 17:07:30

there could be two different vmomi mechanisms in play....

Maximilian Rink

2020-12-02 17:09:47

hmm, from the browser im getting an URI under cls/data/0e4b042c-45bd-4fea-aea1-38e1a5e2c81c/Ubuntu20.04template-1.vmdk on the vcenter

Maximilian Rink

2020-12-02 17:17:59

at least i have some talkingpoints for tomorrow

naadir

2020-12-02 17:18:51

that looks like a content library to me

Maximilian Rink

2020-12-02 17:20:25

itsnt tho. its a packer-created vm that never got uploaded to a content lib.
And there isnt any configured for the cluster or vcenter anyway

Maximilian Rink

2020-12-02 17:20:41

i guess its vmwareautomagic tho

naadir

2020-12-02 17:20:53

yeah, i'm not an expert on the api

Maximilian Rink

2020-12-02 17:22:25

well, i guess i need to file a bug/feature request with packer to get clarification

💯 naadir

cpanato

2020-11-30 16:54:57

@cpanato has joined the channel

cecile

2020-11-30 17:03:12

Vince Prignano (https://kubernetes.slack.com/team/UCD11GCET)

Who's maintaining the image builder project these days?

Original URL: https://kubernetes.slack.com/archives/C8TSNPY4T/p1606754593448800

John Lam

2020-11-30 20:42:26

@John Lam has joined the channel

Sanika Gawhane

2020-12-01 19:32:28

@Sanika Gawhane has joined the channel

Franklin Lee

2020-12-03 23:45:49

@Franklin Lee has joined the channel

cecile

2020-12-04 22:17:18

(will hold until next week)

CecileRobertMichon (https://github.com/CecileRobertMichon)

#448 Update OWNERS

Following yesterday's office hours discussion on adding reviewers, this PR makes the following changes to OWNERS files: • Adds <a href="https://github.com/detiber">@detiber</a> as image-builder approver • Adds <a href="https://github.com/EleanorRigby">@EleanorRigby</a> and <a href="https://github.com/randomvariable">@randomvariable</a> as image-builder reviewers • Adds <code>cluster-api-maintainers</code> as images/capi approvers (and removes <code>cluster-api-admins</code>) • Adds <code>cluster-api-aws-maintainers</code> as images/capi/packer/ami approvers • Adds <code>cluster-api-azure-maintainers</code> as images/capi/packer/azure approvers • Adds <code>cluster-api-gcp-maintainers</code> as images/capi/packer/gce approvers • Adds <code>cluster-api-vsphere-maintainers</code> as images/capi/packer/ova approvers Going forward, if you are interested in becoming an image-builder reviewer and have been helping with PR reviews/made significant contributions to the project, please open a PR to add yourself to the reviewers list (note that you must already be a member of kubernetes-sigs). /assign <a href="https://github.com/akutz">@akutz</a> <a href="https://github.com/codenrhoden">@codenrhoden</a> <a href="https://github.com/detiber">@detiber</a> <a href="https://github.com/figo">@figo</a> <a href="https://github.com/justinsb">@justinsb</a> <a href="https://github.com/luxas">@luxas</a> <a href="https://github.com/moshloop">@moshloop</a> <a href="https://github.com/timothysc">@timothysc</a> /hold to give all maintainers a chance to review

Assignees

justinsb, akutz, figo, timothysc, detiber, moshloop, codenrhoden, luxas

Labels

approved, cncf-cla: yes, do-not-merge/hold, size/M

voor

2020-12-05 15:36:34

Very exciting and we'll deserved across the board

cecile

2020-12-08 21:03:17

Merged. Going forward, if you are interested in becoming an image-builder reviewer and have been helping with PR reviews/made significant contributions to the project, please open a PR to add yourself to the reviewers list (note that you must already be a member of kubernetes-sigs).

👍 cpanato, dongsupark

🎉 jdetiber, codenrhoden

voor

2020-12-08 21:04:21

Yay! 🎉

dongsupark

2020-12-10 07:50:24

Thanks to Travis for approving the Flatcar PR, but the CI failed , and I cannot say /retest. Can anyone with privilege please give it /ok-to-test?

k8s-ci-robot (https://github.com/k8s-ci-robot)

Comment on #248 capi: support Flatcar Container Linux

<a href="https://github.com/dongsupark">@dongsupark</a>: The following test failed, say <code>/retest</code> to rerun all failed tests: <https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=image-builder&pr=248|Full PR test history>. <https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Adongsupark|Your PR dashboard>. Please help us cut down on flakes by <a href="https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests">linking to</a> an <https://github.com/kubernetes-sigs/image-builder/issues?q=is:issue+is:open|open issue> when you hit one in your PR. Instructions for interacting with me using PR comments are available <a href="https://git.k8s.io/community/contributors/guide/pull-requests.md">here</a>. If you have questions or suggestions related to my behavior, please file an issue against the <a href="https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:|kubernetes/test-infra">https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:|kubernetes/test-infra</a> repository. I understand the commands that are listed <a href="https://go.k8s.io/bot-commands">here</a>.

dongsupark

2020-12-10 21:25:12

Thanks @jsturtevant! Now it is merged. 🙂

codenrhoden

2020-12-10 21:29:28

Woohoo! Thanks for hanging in there @dongsupark and seeing this all the way in. It was a lot of work! Conrats!

🎉 jsturtevant, cecile, dongsupark

dongsupark

2020-12-11 08:22:01

Thank you so much for taking care of it, even during your health recovery. 🙂

TAKAHASHI Shuuji

2020-12-12 11:26:56

@TAKAHASHI Shuuji has joined the channel

Arunkumar Venkataramanan (DeepBrainz AI)

2020-12-12 16:02:55

@Arunkumar Venkataramanan (DeepBrainz AI) has joined the channel

hidekazuna

2020-12-16 10:34:09

@dongsupark I proposed PR Flatcar related. Please review

hidekazuna (https://github.com/hidekazuna)

#460 Fix QEMU image for OpenStack not to execute Flatcar specific provisioner and post-processor

bootstrap-flatcar.sh provisioner and vagrant post-processor are only needed for Flatcar. Fixes: <a href="https://github.com/kubernetes-sigs/image-builder/issues/454">#454</a>

Labels

cncf-cla: yes, size/M

Comments

👍 dongsupark

dongsupark

2020-12-16 10:38:02

Thanks, I will have a look.

Norman Bhaskara

2020-12-16 14:11:59

@Norman Bhaskara has joined the channel

Andy Townsend

2020-12-16 14:34:28

is it possible to launch the images built by image-builder without CAPI? If so, how do you get the kubeconfig?

cecile

2020-12-16 16:16:37

You can build VMs using those images on any infrastructure/cloud provider. Are you asking about building k8s clusters though?

Andy Townsend

2020-12-16 16:22:20

yeah, so I built an AMI (that I eventually plan to add some extra provisioners to install our software via helm charts). I can deploy it via CAPI but I'm wondering if we could also share that AMI with a customer so they could just run it in there account.

cecile

2020-12-16 16:30:35

I’m no AWS expert but I’m sure pretty sure there are ways to share AMIs with other users, @naadir might be able to help

Andy Townsend

2020-12-16 16:32:03

yeah, that part isn't an issue. It's more, does that AMI have to be deployed via CAPI. When I deployed the AMI manually and SSH'd in, it didn't seem to have a valid kubeconfig to be able to run kubectl

naadir

2020-12-16 16:36:47

there's no kubeconfig because kubeadm hasn't been run.

naadir

2020-12-16 16:37:12

if all you want to do is have a single node cluster without cluster api involved, you can run kubeadm init on that machine

👍 cecile

naadir

2020-12-16 16:37:30

that drops /etc/kubernetes/admin.conf at the end of the process

cecile

2020-12-16 16:37:33

The AMI itself is just an OS image with preinstalled components and container images. In order to bootstrap the vm into a Kubernetes cluster you need some mechanism such as CAPI (which uses kubeadm underne) or kubeadm to run

Andy Townsend

2020-12-16 16:38:17

Thanks. I just found the kubeadm init stuff and have just ran that now.

Veekay

2020-12-19 07:37:52

@Veekay has joined the channel

Veekay

2020-12-19 09:55:24

Folks
, any method to estimate how much resource should be planned for image runtime require ment

Mike McGhee

2020-12-19 22:24:53

@Mike McGhee has joined the channel

Kritsada Chinpala

2020-12-28 11:39:51

@Kritsada Chinpala has joined the channel

Zespre Chang

2020-12-28 13:26:03

@Zespre Chang has joined the channel

jsturtevant

2021-01-06 21:28:51

I started work for adding goss validation for windows. It required some updates to the goss provisioner. Is there someone to give a ping for a review?

jsturtevant (https://github.com/jsturtevant)

#39 Add support for using goss to target Windows

This provisioner is used to provide goss testing in the <a href="https://image-builder.sigs.k8s.io/capi/goss/goss.html">image-builder project</a>. I recently added <a href="https://github.com/kubernetes-sigs/image-builder/pull/382#issuecomment-707245381">Windows support to image-builder</a> and would like to run the goss test against it during provision in the same manner as the other images. Goss also now has alpha support for <a href="https://github.com/aelsabbahy/goss/blob/master/docs/platform-feature-parity.md">Windows</a>. I've updated the code here to support Windows and added a few tests. Here is the output of it running against a windows VM. It is using a custom goss url while I wait for the fix in goss <a href="https://github.com/aelsabbahy/goss/pull/666">aelsabbahy/goss#666</a> ``` ==> vhd-windows-2019: Provisioning with Goss ==> vhd-windows-2019: Configured to run on Windows vhd-windows-2019: Creating directory: /tmp/goss vhd-windows-2019: vhd-windows-2019: vhd-windows-2019: Directory: C:\tmp vhd-windows-2019: vhd-windows-2019: vhd-windows-2019: Mode LastWriteTime Length Name vhd-windows-2019: ---- ------------- ------ ---- vhd-windows-2019: d----- 12/17/2020 6:37 PM goss vhd-windows-2019: vhd-windows-2019: vhd-windows-2019: Installing Goss from, <a href="https://k8swin.blob.core.windows.net/k8s-windows/goss/goss-alpha-windows-amd64.exe">https://k8swin.blob.core.windows.net/k8s-windows/goss/goss-alpha-windows-amd64.exe</a> vhd-windows-2019: Downloading Goss to /tmp/goss-windows-amd64.exe ==> vhd-windows-2019: % Total % Received % Xferd Average Speed Time Time Time Current ==> vhd-windows-2019: Dload Upload Total Spent Left Speed ==> vhd-windows-2019: 100 11.5M 100 11.5M 0 0 11.5M 0 0:00:01 --:-:-- 0:00:01 26.4M ==> vhd-windows-2019: 'chmod' is not recognized as an internal or external command, ==> vhd-windows-2019: operable program or batch file. ==> vhd-windows-2019: Uploading goss tests... vhd-windows-2019: Uploading vars file packer/goss/goss-vars.yaml vhd-windows-2019: Inline variables are --vars-inline {"ARCH":"amd64","OS":"windows","PROVIDER":"azure","containerdversion":"1.4.3","kubernetescnisourcetype":"","kubernetescniversion":"","kubernetessourcetype":"pkg","kubernetesversion":"v1.17.11"} vhd-windows-2019: Env variables are set "GOSSUSEALPHA=1" && vhd-windows-2019: Uploading Dir packer/goss vhd-windows-2019: Creating directory: /tmp/goss/goss vhd-windows-2019: vhd-windows-2019: vhd-windows-2019: Directory: C:\tmp\goss vhd-windows-2019: vhd-windows-2019: vhd-windows-2019: Mode LastWriteTime Length Name vhd-windows-2019: ---- ------------- ------ ---- vhd-windows-2019: d----- 12/17/2020 6:37 PM goss vhd-windows-2019: vhd-windows-2019: ==> vhd-windows-2019: ==> vhd-windows-2019: ==> vhd-windows-2019: ==> vhd-windows-2019: Running goss tests... ==> vhd-windows-2019: Running GOSS render command: cd /tmp/goss && set "GOSSUSEALPHA=1" && /tmp/goss-windows-amd64.exe --gossfile goss/goss.yaml --vars /tmp/goss/goss-vars.yaml --vars-inline {"ARCH":"amd64","OS":"windows","PROVIDER":"azure","containerdversion":"1.4.3","kubernetescnisourcetype":"","kubernetescniversion":"","kubernetessourcetype":"pkg","kubernetesversion":"v1.17.11"} render > /tmp/goss-spec.yaml ==> vhd-windows-2019: Goss render ran successfully ==> vhd-windows-2019: Running GOSS render debug command: cd /tmp/goss && set "GOSSUSEALPHA=1" && /tmp/goss-windows-amd64.exe --gossfile goss/goss.yaml --vars /tmp/goss/goss-vars.yaml --vars-inline {"ARCH":"amd64","OS":"windows","PROVIDER":"azure","containerdversion":"1.4.3","kubernetescnisourcetype":"","kubernetescniversion":"","kubernetessourcetype":"pkg","kubernetesversion":"v1.17.11"} render -d > /tmp/debug-goss-spec.yaml ==> vhd-windows-2019: Goss render debug ran successfully ==> vhd-windows-2019: Running GOSS validate command: cd /tmp/goss && set "GOSSUSEALPHA=1" && /tmp/goss-windows-amd64.exe --gossfile goss/goss.yaml --vars /tmp/goss/goss-vars.yaml --vars-inline {"ARCH":"amd64","OS":"windows","PROVIDER":"azure","containerdversion":"1.4.3","kubernetescnisourcetype":"","kubernetescniversion":"","kubernetessourcetype":"pkg","kubernetes_version":"v1.17.11"} validate --retry-timeout 0s --sleep 1s -f json -o pretty vhd-windows-2019: { vhd-windows-2019: "results": [ vhd-windows-2019: { vhd-windows-2019: "duration": 1587645100, vhd-windows-2019: "err": null, vhd-windows-2019: "expected": [ vhd-windows-2019: "0" vhd-windows-2019: ], vhd-windows-2019: "found": [ vhd-windows-2019: "0" vhd-windows-2019: ], vhd-windows-2019: "human": "", vhd-windows-2019: "meta": null, vhd-windows-2019: "property": "exit-status", vhd-windows-2019: "resource-id": "kubectl.exe version --short --client=true -o json | powershell -Command \"($input | convertfrom-json).clientVersion.gitVersion\"", vhd-windows-2019: "resource-type": "Command", vhd-windows-2019: "result": 0, vhd-windows-2019: "successful": true, vhd-windows-2019: "summary-line": "Command: kubectl.exe version --short --client=true -o json | powershell -Command \"($input | convertfrom-json).clientVersion.gitVersion\": exit-status: matches expectation: [0]", vhd-windows-2019: "test-type": 0, vhd-windows-2019: "title": "" vhd-windows-2019: }, vhd-windows-2019: { vhd-windows-2019: "duration": 1587645100, vhd-windows-2019: "err": null, vhd-windows-2019: "expected": [ vhd-windows-2019: "0" vhd-windows-2019: ], vhd-windows-2019: "found": [ vhd-windows-2019: "0" vhd-windows-2019: ], vhd-windows-2019: "human": "", vhd-windows-2019: "meta": null, vhd-windows-2019: "property": "exit-status", vhd-windows-2019: "resource-id": "kubeadm.exe version -o json | powershell -Command \"($input | convertfrom-json).clientVersion.gitVersion\"", vhd-windows-2019: "resource-type": "Command", vhd-windows-2019: "result": 0, vhd-windows-2019: "successful": true, vhd-windows-2019: "summary-line": "Command: kubeadm.exe version -o json | powershell -Command \"($input | convertfrom-json).clientVersion.gitVersion\": exit-status: matches expectation: [0]", vhd-windows-2019: "test-type": 0, vhd-windows-2019: "title": "" vhd-windows-2019: }, vhd-windows-2019: { vhd-windows-2019: "duration": 1822111700, vhd-windows-2019: "err": null, vhd-windows-2019: "expected": [ vhd-windows-2019: "0" vhd-windows-2019: ], vhd-windows-2019: "found": [ vhd-windows-2019: "0" vhd-windows-2019: ], vhd-windows-2019: "human": "", vhd-windows-2019: "meta": null, vhd-windows-2019: "property": "exit-status", vhd-windows-2019: "resource-id": "kubelet.exe --version | powershell -command \"$input\".Split(\" \")[1]", vhd-windows-2019: "resource-type": "Command", vhd-windows-2019: "result": 0, vhd-windows-2019: "successful": true, vhd-windows-2019: "summary-line": "Command: kubelet.exe --version | powershell…

cecile

2021-01-11 18:02:53

@Tushar Aggarwal maybe?

farodin91

2021-01-07 16:56:17

@farodin91 has joined the channel

Maximilian Rink

2021-01-08 15:59:12

hi folx, any further input for ?

MaxRink (https://github.com/MaxRink)

#408 Add subiquity-based flavour for Ubuntu 20.04 based upon UEFI

As discussed in the office hours

Labels

cncf-cla: yes, needs-ok-to-test, size/L

Comments

Danilo Tiago

2021-01-11 18:25:29

@Danilo Tiago has joined the channel

Veekay

2021-01-13 09:25:40

Hi Folks any deployment architecture for including dive tool

voor

2021-01-13 11:54:24

dive is packaged as a debian and rpm, you can look at the documentation on how to include additional Debian or rpm packages

Maximilian Rink

2021-01-14 10:19:39

Hi Folx, i would like to hear your opinion on this:
We have this internally and i think it might be useful for other folx as well, especially in the dockerhub ratelimit days 🙂

MaxRink (https://github.com/MaxRink)

#474 [WiP] Expose ContainerD mirror support for "advanced" usecases

Internally we run all our Clusters airgapped, thus we make use of the mirror support in ContainerD. This PR exposes that in the image building process so that other folx can use it as well, if it would be applicable in their environment. This is a WiP, mostly there to gather feedback 😅 ( for example its not working for flatcar and windows right now, as its a straight copy from our fork)

Labels

cncf-cla: yes, do-not-merge/work-in-progress, needs-ok-to-test, size/M

Comments

neolit123

2021-01-14 15:36:03

i can try joining the call today to chat about

randomvariable (https://github.com/randomvariable)

#471 containerd should be using systemd cgroup hierarchy

What steps did you take and what happened: [A clear and concise description on how to REPRODUCE the bug.] As per <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/">https://kubernetes.io/docs/setup/production-environment/container-runtimes/</a>, there should be only be a single cgroup hiearchy running on the OS. We do not configure containerd to run using the systemd hierarchy, such that Kubernetes will have an incorrect view of resource utilisation. What did you expect to happen: ContainerD should be configured to use systemd hierarchy Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] ContainerD config.toml should have something along the lines of <code> [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] ... [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true </code> Environment: Project (<a href="https://github.com/kubernetes-sigs/image-builder/tree/master/images/capi">Image Builder for Cluster API</a>, <a href="https://github.com/kubernetes-sigs/image-builder/tree/master/images/kube-deploy/imagebuilder">kube-deploy/imagebuilder</a>, <a href="https://github.com/kubernetes-sigs/image-builder/tree/master/images/konfigadm">konfigadm</a>): Additional info for Image Builder for Cluster API related issues: • OS (e.g. from <code>/etc/os-release</code>, or <code>cmd /c ver</code>): • Packer Version: • Packer Provider: • Ansible Version: • Cluster-api version (if using): • Kubernetes version: (use <code>kubectl version</code>): /kind bug [One or more /area label. See <a href="https://github.com/kubernetes-sigs/cluster-api/labels?q=area">https://github.com/kubernetes-sigs/cluster-api/labels?q=area</a> for the list of labels]

Labels

kind/bug

Comments

Lauri Apple

2021-01-14 18:15:36

@Lauri Apple has left the channel

Victor Lu

2021-01-16 11:07:56

@Victor Lu has joined the channel

Carlos Arturo Quiroga

2021-01-24 01:22:42

@Carlos Arturo Quiroga has joined the channel

Hina

2021-01-28 18:42:19

@Hina has joined the channel

JSEvans

2021-02-05 12:05:52

@JSEvans has joined the channel

Maximilian Rink

2021-02-09 19:34:27

@codenrhoden has capv shipped with any ubuntu 20.04 image yet? at least officially

codenrhoden

2021-02-09 19:36:11

Nothing I’ve ever pushed. We really should start doing that. 🙂 Nothing stopping it from happening.

Maximilian Rink

2021-02-09 19:36:44

yeah, but that might be the best point to inbtroduce efi builds as @naadir wanted to do a few months ago 😄

asatish

2021-02-12 17:56:55

@asatish has joined the channel

William Lam

2021-02-17 04:52:55

@William Lam has joined the channel

Maximilian Rink

2021-02-18 19:56:16

hmm, we added ovftool as dependency for vsphere builds, but havent updated the dockerfile yet

Maximilian Rink

2021-02-18 19:59:34

And the download is behind a registration wall, so much for "just adding it to the image"

moshloop

2021-02-19 09:36:22

I have some capacity spinning up soon and plan to tackle this + ova python script in the cli

Maximilian Rink

2021-02-19 09:42:43

the EULA of ovatool makes shipping it in the container imposiible on first glance. Am i right @codenrhoden?

Maximilian Rink

2021-02-19 09:43:50

For now, i've settled for this:

ENV LC_CTYPE=POSIX
ENV OVFTOOL_FILENAME=VMware-ovftool-4.4.1-16812187-lin.x86_64.bundle

ADD $OVFTOOL_FILENAME /tmp/

RUN /bin/sh /tmp/$OVFTOOL_FILENAME --console --required --eulas-agreed && <br>    rm -f /tmp/$OVFTOOL_FILENAME

Which requires you to download the bumdle on your own beforehand

Sanika Gawhane

2021-03-26 00:00:04

Hi @Maximilian Rink - did you encounter something like this -

DRoet (https://github.com/DRoet)

#1939 Failed to compute cache key in newer version

This is a docker issue but it seems to be related to BuildKit only. this is something that was still working in docker <code>~19.03.10</code> but stopped functioning in <code>20.10.0+</code>. I managed to bring down my DockerFile to a minimal repro: This works (A.DockerFile): ``` FROM php:7.4.13-cli COPY --from=composer:2.0.8 /usr/bin/composer /usr/local/bin/composer <code> This also works (**B.DockerFile**): </code> FROM php:7.4.13-cli COPY --from=mlocati/php-extension-installer /usr/bin/install-php-extensions /usr/bin/ <code> This no longer works (**C.DockerFile**): </code> FROM php:7.4.13-cli COPY --from=mlocati/php-extension-installer /usr/bin/install-php-extensions /usr/bin/ COPY --from=composer:2.0.8 /usr/bin/composer /usr/local/bin/composer <code> Output from running A and C after eachother: </code> C:\Users\Test>set "DOCKERBUILDKIT=1" & docker build -f A.Dockerfile . [+] Building 3.6s (7/7) FINISHED => [internal] load build definition from A.Dockerfile 0.0s => => transferring dockerfile: 132B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for <a href="http://docker.io/library/php:7.4.13-cli">docker.io/library/php:7.4.13-cli</a> 2.9s => CACHED FROM <a href="http://docker.io/library/composer:2.0.8">docker.io/library/composer:2.0.8</a> 0.0s => => resolve <a href="http://docker.io/library/composer:2.0.8">docker.io/library/composer:2.0.8</a> 0.5s => CACHED [stage-0 1/2] FROM <a href="http://docker.io/library/php:7.4.13_cli@sha256:c099060944167d20100140434ee13b7c134bc53ae8c0a72e81b8f01c07a1f49d">docker.io/library/php:7.4.13cli@sha256:c099060944167d20100140434ee13b7c134bc53ae8c0a72e81b8f01c07a1f49d</a> 0.0s => [stage-0 2/2] COPY --from=composer:2.0.8 /usr/bin/composer /usr/local/bin/composer 0.1s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:ea6d75bc9ad24e800c8083e9ea6b7774f2bd9610cb0e61b3640058c9c7fe34c6 0.0s C:\Users\Test>set "DOCKERBUILDKIT=1" & docker build -f C.Dockerfile . [+] Building 1.0s (8/8) FINISHED => [internal] load build definition from C.Dockerfile 0.0s => => transferring dockerfile: 221B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for <a href="http://docker.io/library/php:7.4.13-cli">docker.io/library/php:7.4.13-cli</a> 0.2s => FROM <a href="http://docker.io/mlocati/php-extension-installer:latest">docker.io/mlocati/php-extension-installer:latest</a> 0.0s => => resolve <a href="http://docker.io/mlocati/php-extension-installer:latest">docker.io/mlocati/php-extension-installer:latest</a> 0.0s => => sha256:ccf3a05d8241580ad9d2a6c884a735bb248e90942ab23e0f8197f851a999ddac 526B / 526B 0.0s => CACHED FROM <a href="http://docker.io/library/composer:2.0.8">docker.io/library/composer:2.0.8</a> 0.0s => [stage-0 1/3] FROM <a href="http://docker.io/library/php:7.4.13_cli@sha256:c099060944167d20100140434ee13b7c134bc53ae8c0a72e81b8f01c07a1f49d">docker.io/library/php:7.4.13cli@sha256:c099060944167d20100140434ee13b7c134bc53ae8c0a72e81b8f01c07a1f49d</a> 0.0s => CACHED [stage-0 2/3] COPY --from=mlocati/php-extension-installer /usr/bin/install-php-extensions /usr/bin/ 0.0s => ERROR [stage-0 3/3] COPY --from=composer:2.0.8 /usr/bin/composer /usr/local/bin/composer 0.0s ------ > [stage-0 3/3] COPY --from=composer:2.0.8 /usr/bin/composer /usr/local/bin/composer: ------ failed to compute cache key: "/usr/bin/composer" not found: not found ``<code> This doesn't happen consistently in my build, sometimes everything builds fine and there are no issues. I'm using windows 10 (20H2) and the latest version of Docker Desktop that includes</code>Docker version 20.10.2, build 2291f61`, but I have also seen this happen on Linux with the same version

Labels

needs-investigation

Comments

Maximilian Rink

2021-02-19 17:32:07

And am i the only one that noticed the the OVAs dont have the systemd-timesyncd service enabled?

voor

2021-02-19 20:46:26

Hmm, I see extrarepos and I see extradebs, but I don't see a way to add extra keys. 😞

voor

2021-02-19 21:25:53

Docker build is failing with the following:

#16 132.9 hack/ensure-ovftool.sh
#16 132.9 ovftool must be present to build OVAs. If already installed
#16 132.9 make sure to add it to the PATH env var. If not installed, please
#16 132.9 install latest from .
#16 132.9 make: * [Makefile:90: deps-ova] Error 1
------
executor failed running [/bin/sh -c make deps]: exit code: 2
make: * [Makefile:592: docker-build] Error 1

John Lafata

2021-03-03 18:55:07

@John Lafata has joined the channel

fabrizio.pandini

2021-03-04 13:35:22

@fabrizio.pandini has joined the channel

fabrizio.pandini

2021-03-04 13:39:03

PSA Changes for cgroup driver are landing in kubeadm and in CAPI; that’s means that now MUST be implemented before shipping 1.21 images.
Please reach out if you see problems in this …

codenrhoden

2021-03-05 16:39:51

Thanks Fabrizio. I still haven’t been able to move past the issues I was showing before. I’m putting the config file in place as expected, but containerd status doesn’t show that the change has been picked up. I probably need to pair with someone early next week to work through it. I know this needs to happen.

fabrizio.pandini

2021-03-05 19:11:11

thank_you

codenrhoden

2021-03-08 16:58:51

No luck on my side with this. same issues with the config change not taking effect. I’d like to pair with @neolit123 when possible to figure it out.

Maximilian Rink

2021-03-05 17:32:28

FYI: we have an containerd CVE
1.4.4 fixes it, so you propably want to bump that before doing a tag @codenrhoden

❤️ Dell R

codenrhoden

2021-03-05 17:34:13

Thanks. Sounds like a good idea

codenrhoden

2021-03-05 17:54:23

containerd bump:

codenrhoden (https://github.com/codenrhoden)

#539 bump containerd to 1.4.4

What this PR does / why we need it: This addresses a containerd <a href="https://github.com/containerd/containerd/security/advisories/GHSA-6g2q-w5j3-fwh4">CVE</a>

Labels

approved, cncf-cla: yes, size/XS

Comments

Jesse Hu

2021-03-07 03:28:41

@Jesse Hu has joined the channel

jsturtevant

2021-03-09 19:05:31

@codenrhoden @cecile

jsturtevant (https://github.com/jsturtevant)

Comment on #368 alphabetical sort of all json files

> Get-Service : Cannot find any service with service name 'WindowsAzureTelemetryService'. Is a not causing the builds to fail. That windows service can sometimes take a time to come up when doing the sysprep step but if you look directly afterwards the it completes with <code>�[0;32m vhd-windows-2019: Using cloudbase-init unattend file for sysprep: C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\Unattend.xml�[0m �[0;32m vhd-windows-2019: IMAGE_STATE_COMPLETE�[0m</code> The few jobs in <a href="https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-azure-vhds">https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-azure-vhds</a> that are failing look to be network related. Since this started in the last 12hrs It might be due to the code freeze happening today it putting extra pressure on Prow. For instance, these two look like they were related to pods being evicted • <a href="https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/535/pull-azure-vhds/1369222230629683200">https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigsimage-builder/535/pull-azure-vhds/1369222230629683200</a> • <a href="https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/535/pull-azure-vhds/1369217728316968960">https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigsimage-builder/535/pull-azure-vhds/1369217728316968960</a> There does seem to be some network timeouts with Windows ansbile here, which may also be related • <a href="https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/535/pull-azure-vhds/1369249994426552320">https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigsimage-builder/535/pull-azure-vhds/1369249994426552320</a> • <a href="https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/535/pull-azure-vhds/1369223292774256640">https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigsimage-builder/535/pull-azure-vhds/1369223292774256640</a> And network issues retreiving the storage accounts here: • <a href="https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/368/pull-azure-vhds/1369040674711670784">https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/368/pull-azure-vhds/1369040674711670784</a>

jsturtevant

2021-03-09 19:05:53

it looks like there are Prow flakes happening with regards to networking

codenrhoden

2021-03-09 19:06:46

Thanks for the explanation there… 🤞 yay code freeze!

jsturtevant

2021-03-09 19:08:19

yea, Im not seeing anything in particular.... this one couldn't get ansible:

jsturtevant

2021-03-09 19:08:37

jsturtevant

2021-03-09 19:08:41

is the right link

jsturtevant

2021-03-09 19:09:28

Not sure if there is much we can do on those?

codenrhoden

2021-03-09 19:10:43

doesn’t seem like it

Zakarias

2021-03-10 14:57:20

@Zakarias has joined the channel

Ben Cressey

2021-03-10 18:15:17

@Ben Cressey has joined the channel

jayunit100-pub-chnl-plz

2021-03-11 17:00:54

@jayunit100-pub-chnl-plz has joined the channel

jayunit100-pub-chnl-plz

2021-03-11 17:05:14

<-- hi folks, windows is broke :)

jayunit100 (https://github.com/jayunit100)

#544 windows broken ova build

What steps did you take and what happened: windows is broke now (recent typed version additions)

codenrhoden

2021-03-11 17:06:50

we’ll have to see if we can find a way to do some sort of Windows CI.

jsturtevant

2021-03-11 17:08:54

we have azure windows ci... is there a way to add OVA?

codenrhoden

2021-03-11 17:52:21

We’ve been working on getting the OVA stuff in place. Still running into basic issues with Packer over a VPN connection between Prow and the cloud provider. Working on it though!

👍 jsturtevant, jayunit100-pub-chnl-plz

Sherif Abdel-Naby

2021-03-13 17:32:44

@Sherif Abdel-Naby has joined the channel

jsturtevant

2021-03-16 20:48:42

@codenrhoden @cecile I opened a pr to mitigate the failures we were seeing:

jsturtevant (https://github.com/jsturtevant)

#550 Update windows defender to be best effort

What this PR does / why we need it: Make windows defender best effort during build. Defender is running on the host so the updates will be run periodically once the machine is running. Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes <a href="https://github.com/kubernetes-sigs/image-builder/issues/549">#549</a> Additional context Add any other context for the reviewers I am going to connect with the Windows defender team to figure out why it might be failing occasionally and will follow up.

Labels

cncf-cla: yes, size/S

Comments

:thank_you: codenrhoden

asatish

2021-03-17 15:47:18

hi, I am making changes to image-builder to support RHEL8 for vsphere ISO and I can send a PR for it. The VMDK is created and in the step of creating OVF and OVA, images/capi/hack/image-build-ova.py expects to have an entry for rhel8-64 in OSidmap which is not present. I do not know the OSID and version which is needed by vmware. Can someone please help with this info?
OSidmap = {“vmware-photon-64”: {“id”: “36”, “version”: “”, “type”: “vmwarePhoton64Guest”},
“centos7-64": {“id”: “107", “version”: “7", “type”: “centos7-64"},
“rhel7-64”: {“id”: “80”, “version”: “7”, “type”: “rhel764guest”},
“ubuntu-64": {“id”: “94", “version”: “”, “type”: “ubuntu64Guest”},
“Windows2019Server-64”: {“id”: “112”, “version”: “”, “type”: “windows9srv-64”},
“Windows2004Server-64": {“id”: “112", “version”: “”, “type”: “windows9srv-64"}}

for the time being I added this line, script was able to move ahead, but I would like to know the correct id/version/type for rhel8
“rhel8-64”: {“id”: “80”, “version”: “8”, “type”: “rhel864guest”},

Sean Smith

2021-03-18 18:28:23

@Sean Smith has joined the channel

Maximilian Rink

2021-03-19 17:42:01

we should propably set for the OVAs, right?

docs/book/known_issues.md

<pre><code># Known Issues ## VMTools Nice/Device Filtering A number of [CNI](<https://github.com/containernetworking/cni>) implementations (such Calico, Antrea, and etc) introduce networking artifacts that interfere with the normal operation of vSphere's internal reporting for network/device interfaces. To address this issue, an `exclude-nics` filter for VMTools needs to be applied in order to prevent these artifacts from getting reported to vSphere and causing problems with network/device associations to vNICs on virtual machines. The recommended `exclude-nics` filter is as follows for `/etc/vmware-tools/tools.conf`: ```bash [guestinfo] primary-nics=eth0 exclude-nics=antrea-**,cali**,ovs-system,br**,flannel**,veth**,docker**,virbr**,vxlan_sys_**,genev_sys_**,gre_sys_**,stt_sys_**,????????-?????? </pre> Each filter represents known CNI network/device interfaces. Most filters are straight foward, such as <code>docker**</code> for devices based on docker. Some filters, such as <code>????????-??????</code>, aren't so straight-forward as that filter identifies Antrea devices which get created per POD. Restart VMTools for the changes to take effect. <code>bash /etc/vmware-tools/services.sh start /etc/vmware-tools/services.sh stop /etc/vmware-tools/services.sh restart </code> ```

Mateusz Gozdek (invidian)

2021-03-19 18:39:15

@Mateusz Gozdek (invidian) has joined the channel

Mateusz Gozdek (invidian)

2021-03-19 18:45:34

👋 I'm trying to add Flatcar support to image-builder for Azure and I have some questions:

Are VHD images required? Or are only SIG images sufficient? I've seen that VHD images are deprecated and actually building Flatcar images out of Marketplace images is not possible due to some limitations around unmanaged disks uage.

I'm stuck at the following error message: the Shared Gallery Image to which to publish the managed image version to does not exist in the resource group mat-dev. I can't quite figure out what I'm doing wrong.

jsturtevant

2021-03-19 19:52:52

We use the VHD's for cluster-api-azure for the sample images since sigs are do not allow public images. If there is limitations I believe it would be find to do only sigs but at this we wouldn't be able to add automated tests to cluster-api-azure I think.

jsturtevant

2021-03-19 19:53:45

for #2 did you modify to include new flatcar definition?

images/capi/packer/azure/scripts/init-sig.sh:50

<pre><code>az sig image-definition create \ </code></pre>

jsturtevant

2021-03-19 19:55:16

I believe we just added gen2 support that only went to sigs for a reference:

jsturtevant

2021-03-19 19:55:31

alexeldeib (https://github.com/alexeldeib)

#422 add support for Azure Gen2 VMs

Assignees

mboersma, jsturtevant, CecileRobertMichon

Labels

approved, cncf-cla: yes, lgtm, size/M

Mateusz Gozdek (invidian)

2021-03-19 19:56:58

We use the VHD's for cluster-api-azure for the sample images since sigs are do not allow public images. If there is limitations I believe it would be find to do only sigs but at this we wouldn't be able to add automated tests to cluster-api-azure I think.

Thanks, that make sense. I'll work on making VHD's to work.

Mateusz Gozdek (invidian)

2021-03-19 19:57:16

2 I already figured out, though It's quite tricky with packer to get all parameters right

👍 jsturtevant

bavarianbidi

2021-03-22 13:30:01

@bavarianbidi has joined the channel

jdetiber

2021-03-23 16:32:03

Has anyone else noticed any issues with recent capi image-builder images related to cloud-config and cloud-final services not running on Ubuntu 18.04 with recent builds?

naadir

2021-03-23 16:33:18

not again

jdetiber

2021-03-23 16:33:47

well, I'm not quite sure if it's an issue of my own making with the raw image builder PR 🙂

cecile

2021-03-23 18:35:26

Didn’t run into any issues when we built new k8s images this week for azure

jdetiber

2021-03-23 18:36:35

Yeah, I figured out the issue, since I'm trying to use the ec2 datasource in an unknown environment I needed to lay down a config file in /etc/cloud-ds-identify.cfg so that the systemd generator would do the right thing...

Gurpreet singh

2021-03-26 04:27:00

@Gurpreet singh has joined the channel

Vignesh Goutham

2021-03-26 17:27:06

@Vignesh Goutham has joined the channel

Vignesh Goutham

2021-03-26 17:34:50

Hello guys 👋, I’m trying to do a capi image build for aws ami. It looks like the ansible errors out saying it needs to be root to perform yum commands on the remote ec2-instance. The interesting part is the build works completely fine on mac (local) and also on an ec2-instance running amazon linux 2. It only errors out when run on amazon linux 2 container.
I created this issue as well with logs -
I’d appreciate it, if anyone has ideas to go about debugging this.

vignesh-goutham (https://github.com/vignesh-goutham)

#565 Image builder for ami amazon 2 fails with need to be root on tasks

What steps did you take and what happened: While trying to build an ami for amazon 2, I ran the make targets <code>make deps-ami</code> and <code>make build-amazon-2</code> make deps successfully installs ansible and all the other dependencies. make build-amazon-2 fails while trying to execute the "TASK [setup : add epel repo]" Logs: ``` bash-4.3# make build-ami-amazon-2 hack/ensure-ansible.sh hack/ensure-ansible-windows.sh hack/ensure-packer.sh hack/ensure-goss.sh Right version of binary present packer build -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/config/kubernetes.json" -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/config/cni.json" -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/config/containerd.json" -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/config/ansible-args.json" -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/config/goss-args.json" -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/config/common.json" -color=true -var-file="/go/src/github.com/aws/eks-distro/image-builder/images/capi/packer/ami/amazon-2.json" packer/ami/packer.json amazon-2: output will be in this color. ==> amazon-2: Prevalidating any provided VPC information ==> amazon-2: Prevalidating AMI Name: capa-ami-amazon-2--1616* amazon-2: Found Image ID: ami-05b622b5fa0269787 ==> amazon-2: Creating temporary keypair: packer_* ==> amazon-2: Creating temporary security group for this instance: packer****** ==> amazon-2: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups... ==> amazon-2: Launching a source AWS instance... ==> amazon-2: Adding tags to source instance amazon-2: Adding tag: "Name": "Packer Builder" amazon-2: Instance ID: i-**** ==> amazon-2: Waiting for instance (i-****) to become ready... ==> amazon-2: Using ssh communicator to connect: ****** ==> amazon-2: Waiting for SSH to become available... ==> amazon-2: Connected to SSH! ==> amazon-2: Provisioning with shell script: /tmp/packer-shell347273043 ==> amazon-2: Provisioning with shell script: ./packer/files/bootstrap-flatcar.sh ==> amazon-2: Provisioning with Ansible... amazon-2: Setting up proxy adapter for Ansible.... ==> amazon-2: Executing Ansible: ansible-playbook -e packerbuildname=amazon-2 -e packerbuildertype=amazon-ebs -e packerhttpaddr=ERRHTTPADDRNOTIMPLEMENTEDBYBUILDER --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerdurl=<a href="https://github.com/containerd/containerd/releases/download/v1.4.3/cri-containerd-cni-1.4.3-linux-amd64.tar.gz">https://github.com/containerd/containerd/releases/download/v1.4.3/cri-containerd-cni-1.4.3-linux-amd64.tar.gz</a> containerdsha256=2697a342e3477c211ab48313e259fd7e32ad1f5ded19320e6a559f50a82bff3d ...... ****** (hidden) amazon-2: amazon-2: PLAY [all] ********************************************** amazon-2: amazon-2: TASK [Gathering Facts] ************************************** amazon-2: [WARNING]: Platform linux on host default is using the discovered Python amazon-2: ok: [default] amazon-2: interpreter at /usr/bin/python, but future installation of another Python amazon-2: interpreter could change the meaning of that path. See <a href="https://docs.ansible.com">https://docs.ansible.com</a> amazon-2: /ansible/2.10/referenceappendices/interpreterdiscovery.html for more amazon-2: information. amazon-2: amazon-2: TASK [includerole : node] ** amazon-2: amazon-2: TASK [setup : Put templated sources.list in place] amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : Put templated apt.conf.d/90proxy in place when defined] ** amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : Find existing repo files] ** amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : Disable repos] * amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : Install extra repos] amazon-2: amazon-2: TASK [setup : perform a dist-upgrade] amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : install baseline dependencies] * amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : install extra debs] amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : install pinned debs] * amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : Fetch /etc/os-release] * amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : includetasks] ********************************** amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : RHEL subscription] ******************************** amazon-2: skipping: [default] amazon-2: amazon-2: TASK [setup : add epel repo] **************************** amazon-2: fatal: [default]: FAILED! => {"changed": false, "changes": {"installed": ["/tmp/.ansible/ansible-tmp-1616698577.5169854-2582-419570803580/epel-release-latest-7.noarchImS9dl.rpm"]}, "msg": "You need to be root to perform this command.\n", "rc": 1, "results": ["Loaded plugins: extrassuggestions, langpacks, priorities, update-motd\n"]} amazon-2: amazon-2: PLAY RECAP ** amazon-2: default : ok=1 changed=0 unreachable=0 failed=1 skipped=12 rescued=0 ignored=0 amazon-2: ==> amazon-2: Provisioning step had errors: Running the cleanup provisioner, if present... ==> amazon-2: Terminating the source AWS instance... ==> amazon-2: Cleaning up any extra volumes... ==> amazon-2: No volumes to clean up, skipping ==> amazon-2: Deleting temporary security group... ==> amazon-2: Deleting temporary keypair... Build 'amazon-2' errored: Error executing Ansible: Non-zero exit status: exit status 2 ==> Some builds didn't complete successfully and had errors: --> amazon-2: Error executing Ansible: Non-zero exit status: exit status 2 ==> Builds finished but no artifacts were created. make: ** [build-ami-amazon-2] Error 1 ``<code> **What did you expect to happen:** The expectation was to complete executing the task and produce an ami in my aws account **Environment:** This bug only occurs in the AL2 container image. I tried running the command on my mac, it passes successfully there. I cross verified env variables, and also checked ansible version. They all seem to match. Project (<a href="https://github.com/kubernetes-sigs/image-builder/tree/master/images/capi">Image Builder for Cluster API</a> Additional info for Image Builder for Cluster API related issues: • OS (e.g. from</code>/etc/os-release<code>, or</code>cmd /c ver`): Amazon Linux 2 (container image) • Packer Version: 1.6.0 • Packer Provider: hashicorp • Ansible Version: 2.10.7 /kind bug [One or more /area label. See <a href="https://github.com/kubernetes-sigs/cluster-api/labels?q=area">https://github.com/kubernetes-sigs/cluster-api/labels?q=area</a> for the list of labels]

Labels

kind/bug

Gabriel Silva

2021-03-26 19:01:22

@Gabriel Silva has joined the channel

Sean Smith

2021-03-29 21:14:31

Hey I’m curious if others are using Image Builder like I am, or I’m doing this in a “bad” way. Currently I’ve got a repo which has our ansible customizations and has image-builder as a git submodule (pinned to a specific tag), my CI pipeline copies the ansible customization into the ansible directory (since they aren’t open source/public) and i’m using the makefile based on the image builder book.

jdetiber

2021-04-01 14:23:01

Instead of using it as a git suubmodule, could use use the published container image and bind mount in your customizations?

👍 Sean Smith

Sean Smith

2021-04-01 18:09:10

I could, that makes sense, I didn’t realize that was an option, I’ll lookup the images and start using them.

Sean Smith

2021-04-01 18:16:15

ok yeah that will work perfectly, just need to copy around some files in the CI pipeline because I can’t chose where my repo is mounted

Amit Mishra

2021-04-01 14:00:02

@Amit Mishra has joined the channel

jayunit100-pub-chnl-plz

2021-04-01 16:04:17

anyone goin to office hrs

jsturtevant

2021-04-01 16:06:21

it's next week

jsturtevant

2021-04-01 16:06:33

happens every 2wks

jsturtevant

2021-04-01 16:08:53

https://calendar.google.com/calendar/u/0/r/week/2021/4/8?eid=Y3I5dHJwbmVucjVvcWEybXBnO[…]0MDhUMTUwMDAwWiBjYWxlbmRhckBrdWJlcm5ldGVzLmlv&pli=1&sf=true Is a link I think

accounts.google.com

Google Calendar

With Google's free online calendar, it’s easy to keep track of life’s important events all in one place.

Original URL: https://calendar.google.com/calendar/u/0/r/week/2021/4/8?eid=Y3I5dHJwbmVucjVvcWEybXBnOTBoZWpiZWNfMjAyMTA0MDhUMTUwMDAwWiBjYWxlbmRhckBrdWJlcm5ldGVzLmlv&pli=1&sf=true

Kubernetes Moderator Service

2021-04-01 17:33:09

@Kubernetes Moderator Service has joined the channel

Kubernetes Moderator Service

2021-04-01 17:33:09

@Kubernetes Moderator Service has joined the channel

jayunit100-pub-chnl-plz

2021-04-06 21:44:14

@codenrhoden any objection to a hack/serve.py script just so we have a canonical example of how to serve artifacts on a url in image-builder?

jayunit100-pub-chnl-plz

2021-04-06 21:47:50

re:

jayunit100 (https://github.com/jayunit100)

#571 imgpkg / bundle / http facade awareness for image builds

Is your feature request related to a problem? Please describe. For windows nodes, you need to build an OVA from scratch - and can't borrow OVAs from elsewhere. Thus anyone using CAPI w/ windows need to use image builder to create their images, for their nodes. This means that the UX around how binaries are pulled into image builder needs to be dumbed down a little, since its not just cloud providers building image builder into their internal workflows - but rather - end users wanting to get a cluster up quickly. problem :when we build windows images, we pull artifacts from a URL, we force users to host their windows kubelet/kubeproxy/containerd/hcsshim/... from an endpoint. USers may not have a good way to host this stuff or may not have an internal process that allows for this. • ... One really clean and scalable way to host files as docker images is <a href="https://carvel.dev/imgpkg/">https://carvel.dev/imgpkg/</a>, which suggests an automatic mechanism for creating a hosted package of imagbuilder windows binaries.... • another option would be to make imagebuilder capable of reading files from a local disk somehow, using something like scp to put them onto an image However, imgpkg doesnt "serve" the files on an endpoint, but rather, directly copies them as a folder to a destination. So, i would like a way to consume from files, rather then a url.... Describe the solution you'd like 1. Can we make it so that an <code>imgpkg</code> aware hosting command in image builder, i.e. one which runs: ``` imgpkg pull -i <a href="http://index.docker.io/periismyfriend/windowz:v1">index.docker.io/periismyfriend/windowz:v1</a> --output /tmp/my-windows-exes python3 -m http.server ``<code> for users, that way we can give folks a very clean image building experience, wherein they only need to provide a docker registry of a</code>imgpkg` repo as the input to image builder. This would mean users dont need to host there own windows executables on a port... instead, the VM that is building the image could just pull down its bits from the imagebuilder node. 2. IF we're not able to bundle imgpkg awareness, can we modify the way we invoke imgbuilder so that it copys executables from its local disk to the ova somehow , rather then doing a pull from a URL ? Describe alternatives you've considered Make a third artifact that connects imgpkg artifacts with imagebuilder execution, i.e. a init container which ran image builder in a pod, pulled the artifacts into a PV/hostpath , and invoked image builder on an endpoint . this however would only work in k8s Additional context /kind feature

Labels

kind/feature

Comments

codenrhoden

2021-04-06 21:56:35

I think that could be useful, yeah. If people are really enterprising, it can also be a way to speed up local builds that are purely upstream. I image it ends up just being a wrapper around the python http serve module?

🎉 jayunit100-pub-chnl-plz

jayunit100-pub-chnl-plz

2021-04-06 21:58:01

yeah , exactly, not anything special but something to make bikeshedding less dangerous bc theres a canonical implementation to borrow

jayunit100-pub-chnl-plz

2021-04-06 21:58:43

want to make clear my intentions here :) goal not to solve all problems but just to say "hey, heres an example" so we can get away from abstract descriptions of something that can be done in like 4 lines of code :)

👍 codenrhoden

jayunit100-pub-chnl-plz

2021-04-06 21:50:29

(it basically is a way for anyone downstream to have a minimal impl of an artifact server) so that theres an end to end example upstream of how to use image builder w/ custom inputs

vrabbi

2021-04-08 13:11:13

Is there a way to run the image builder on an existing machine? I have a need to add a baremetal server to a capv cluster and need to prepare the node. Has any done something like this or any guidance on how to extract the relevant parts of image builder in order to prepare the node correctly?

Peri

2021-04-08 16:41:37

@jsturtevant sorry it was yours!! 😄 this has some notes on running ansible on its own

docs/book/src/capi/windows/windows.md

<pre><code># Windows ## Configuration The `images/capi/packer/config/windows` directory includes several JSON files that define the default configuration for the Windows images: | File | Description | |\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\--| | `packer/config/windows/ansible-args.json` | A common set of variables that are sent to the Ansible playbook | | `packer/config/windows/cloudbase-init.json` | The version of [Cloudbase Init](<https://github.com/cloudbase/cloudbase-init>) to install | | `packer/config/windows/common.json` | Settings for things like which runtime (Docker or Containerd), pause image and other configuration | | `packer/config/windows/kubernetes.json` | The version of Kubernetes to install and it's install path | | `packer/config/windows/containerd.json` | The version of containerd to install | ## Service Manager Image-builder provides you two ways to configure windows services. The default is setup using [nssm](<https://nssm.cc/>) which configures a windows service for kubelet by running `{{ kubernetes_install_path }}\StartKubelet.ps1` allowing easy editing of command arguments in the startup file. The alternate is to use the windows native [sc.exe](<https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/sc-config>) which uses the kubelet argument `--windows-service` to install kubelet as a native windows service with the command line arguments configured on the service. Nssm handles service restarts, if you are using sc.exe you may wish to configure the service restart options on kubelet. To avoid starting kubelet to early, image-builder sets the kubelet service to **manual** which you should consider changing once the node has joined a cluster. ****Important: sc.exe does not support kubeadm KUBELET_KUBEADM_ARGS which is used by Cluster API to pass extra user args**** ## Wins (by rancher) As a workaround for the lack of privileged containers on windows nodes, SIG Windows utilise [Wins](<https://github.com/rancher/wins/>) to allow pods to run processes on hosts, an example is when you are using kube-proxy or a cni provider in a pod. This means that by default we install wins using image-builder. This may be undesirable, if you do not need pod to have the ability to run processes on the host. Additionally, Alpha support for Windows [Host Processes](<https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/1981-windows-privileged-container-support>), which is the windows equivalent for privileged containers, will be introduced in 1.22 which negates the need for wins entirely. To skip installing wins on the host you can add `"wins_url": ""` to your variables. When using containerd, due to lack of host networking, there is currently no way to run cni inside a pod. To work around this, containerd needs to be installed with an additional custom Ansible module or after the image has been created. With CAPI, the addtional configuration can be done with pre/postKubeadm scripts when the node is created. ## Windows Updates When building Windows images it is necessary to install OS and Security updates. Image Builder provides two variables to allow choosing which updates get installed which can be used together or seperately (with individual KBs installed first). To specify the update categories to check, provide a value for `windows_updates_categories` in `packer/config/windows/common.json`. Example: Install all available updates from **all categories**. `"windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups"` Published Cloud Provider images such as Azure or AWS are regularly updated so it may be preferable to specify individual patches to install. This can be achieved by specifying the KB numbers of required updates. To choose individual updates, provide a value for `windows_updates_kbs` in `packer/config/windows/common.json`. Example: `"windows_updates_kbs": "KB4580390 KB4471332"`. ## Using the Ansible Scripts directly Ansible doesn't run on directly on Windows (wsl works) but can used to configure a remote Windows host. For faster development you can create a VM and run Ansible against the Windows VM directly with out using packer. This document gives the high level steps to use Ansible from Linux machine. ## Set up Windows machine Follow the documentation for configuring WinRm on the Windows machine: <https://docs.ansible.com/ansible/latest/user_guide/windows_setup.html#winrm-setup>. Note the [ConfigureRemotingForAnsible.ps1](<https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1>) is for development only. Refer to [Ansible WinRM documentation](<https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html>) for details for advance configuration. After WinRM is installed you can edit or `/etc/ansible/hosts` file with the following: ``` [winhost] &lt;windows ip&gt; [winhost:vars] ansible_user=username ansible_password=&lt;your password&gt; ansible_connection=winrm ansible_winrm_server_cert_validation=ignore </pre> Then run: <code>ansible-playbook -vvv node_windows.yml --extra-vars "@example.vars.yml</code> <h2>MacOS with ansible</h2> The Winrm connection plugin for Ansible on MacOS causes connection issues which can result in <code>ERROR! A worker was found in a dead state</code>. See <a href="https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html#what-is-winrm">https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html#what-is-winrm</a> for more details. To fix the issue on MacOS is to set the no_proxy environment variable. Example: <code> 'no_proxy=** make build-azure-vhd-windows-2019' </code> ```

😄 jsturtevant

Peri

2021-04-08 16:41:43

cc @codenrhoden

vrabbi

2021-04-08 16:56:37

Awesome thanks!

jsturtevant

2021-04-08 17:04:56

note that it is windows specif but its the same general idea

Peri

2021-04-08 16:31:39

@Peri has joined the channel

vrabbi

2021-04-12 10:12:40

I see that in the url.yaml ansible task for downloading k8s binaries no retry logic has been added however for other downloads like for k8s images there is retry logic. Is there a reason for this? Would this be possible to add. If this is something that others would find helpful i can create a pr with the suggested change

Peri

2021-04-16 08:55:17

i dont think there is any particular logic about that it is likely just who made the tasks, maybe put an issue in to track this, it seems like a sensible ask to me

👍 vrabbi

vrabbi

2021-04-16 09:11:23

https://github.com/kubernetes-sigs/image-builder/issues/595

vrabbi (https://github.com/vrabbi)

#595 Add retry logic for kubernetes binary url based downloads

Is your feature request related to a problem? Please describe. When using the url.yaml method for installijg k8s binaries on images no retry logic is present. It can be very common to have sporadic network or server issues which make a download fail. Adding retry logic similar to that in k8s component containers task would be very helpful and would limit the failed builds which i have encountered multiple times. Describe the solution you'd like Add retry logic to all the download tasks in url.yaml playbook Describe alternatives you've considered I have added locally retry logic and that works for me now but i think this should be done in the upstream project. Additional context I am using tkg and therefore need to use the url based downloads as that is the supported method with tkg. /kind feature

Labels

kind/feature

Peri

2021-04-16 09:11:53

perfect thanks @vrabbi!

Nate

2021-04-28 02:31:13

@Nate has joined the channel

chrisg

2021-04-29 14:31:45

@chrisg has joined the channel

Subhash

2021-04-30 09:04:06

@Subhash has joined the channel

Nate

2021-04-30 21:52:00

For vsphere images, is there a packer config for ova disk size? The templates are creating with a 20Gi disk. I can't find where to increase that.

voor

2021-05-02 00:03:52

The base image is just for the ova, when actual clusters are stood up you can change the cloned images disk size

Nate

2021-05-02 00:08:00

ok, that's what I figured but am not seeing something correctly with the provision. have set disk size in cluster config yaml, but seems not to take. probable user error.

voor

2021-05-02 12:01:57

Where are you setting the value?

Nate

2021-05-03 02:28:45

in the cluster config yam via VSphereMachineTemplate:

549-kind: VSphereMachineTemplate
1578-metadata:
1588-  name: dev-cluster
1608-  namespace: dev
1625-spec:
1631-  template:
1643-    spec:
1653-      cloneMode: linkedClone
1682-      datacenter: natelab
1708-      datastore: esxi-local-1
1738:      diskGiB: 75

jsturtevant

2021-05-03 17:08:06

@Peri we increase the disk size for Windows I think?

Peri

2021-05-03 18:01:08

Yep we have 80gb I think?

Peri

2021-05-03 18:01:25

There is a way to change it in image builder too

Nate

2021-05-03 22:14:00

that was my initial thought, but couldn't figure out how to set it with image builder. but i am only using the available base and clone images and adding a deb with image builder.

Peri

2021-05-06 15:16:42

perithompson (https://github.com/perithompson)

#498 Added a variable to configure hard disk size

What this PR does / why we need it: Add the ability to set a variable to define the disk size of ova files. Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issuenumber>, ...) format, will close the issue(s) when PR gets merged): Fixes <a href="https://github.com/kubernetes-sigs/image-builder/issues/428">#428</a> Additional context I have set defaults to the initial values 20GB for Linux and 80GB for Windows but this can be overridden using PACKERFLAGS=-var=disk_size="40926" (size is in MB)

Assignees

codenrhoden, kkeshavamurthy

Labels

approved, cncf-cla: yes, lgtm, ok-to-test, size/M

:1000: Nate

Nate

2021-05-01 05:50:32

are there any plans to allow a template file to pull in modules/templates? sort of like a kustomization patch pattern.

Nate

2021-05-01 19:25:50

i suppose I was looking at the template purpose from the wrong angle. use template for bare min cni and csi, then use post provisioning stage with kustomize for the rest.

voor

2021-05-02 00:02:54

For things like cni and csi I would look at cluster resource sets or something like kapp-controller

voor

2021-05-02 00:03:07

Those shouldn't be anywhere close to image building

Nate

2021-05-02 00:06:35

ah, yes, that is what I'm using. I didn't post this in the right channel. meant to post to base cluster-api.

Maximilian Rink

2021-05-04 09:33:18

containerd 1.5 is out
How long do we want to stick with 1.4.x?

Maximilian Rink

2021-05-04 09:34:19

registry.config_path
looks promising and should make exposing the mirror settings easier.

Scott Carrier

2021-05-05 10:58:33

@Scott Carrier has joined the channel

Chris Privitere

2021-05-05 16:25:02

@Chris Privitere has joined the channel

Chris Privitere

2021-05-05 16:26:41

Is there a way to disable exports of the ova file with image-builder? Image builder is the preferred way to build images for things like TKG downstream but in that scenario there's actually no need to have a .ova file created. The template on the vsphere cluster that packer creates is sufficient. But i cannot for the ilfe of me figure out a way to overload the export config with a later .json file. Any ideas?

codenrhoden

2021-05-25 17:26:19

This is pretty much a limitation with out we have our Packer config files structured at the moment. But I am in full agreement with you:

codenrhoden (https://github.com/codenrhoden)

Comment on #468 Add upload support for ova into vcenter

/remove-lifecycle stale I'd definitely like to keep this issue around... Ultimately I'd like it if this step wasn't necessary -- if we are just putting the OVA back into the same vCenter that built the machine image anyways, is there a way to create the template without having to download the disk, and then re-upload the final OVA? It seems the answer is yes, but not sure at this point how easy that is to accomplish with the current Packer post-processor tooling. Ultimately what we need/want in vCenter is a template (for consumption by CAPV), which Packer can create (<code>"convert_to_template"': "true"</code>), but we also need to be able to set all the vApp properties that are currently set in the OVF XML -- that's what is missing. There are vCenter APIs to do this, but this missing link is how/what to call to do so. But with that in place, all of the steps of downloading the VMDK, creating an OVF/OVA, and then re-uploading become unnecessary.

outofmemory

2021-05-06 03:43:54

@outofmemory has joined the channel

Loty Gero

2021-05-12 00:07:31

@Loty Gero has joined the channel

J.P. Poveda

2021-05-12 21:55:39

@J.P. Poveda has joined the channel

JorgeArteiro

2021-05-14 11:33:25

@JorgeArteiro has joined the channel

Ulan Manas

2021-05-14 12:48:07

@Ulan Manas has joined the channel

Hoon Jo

2021-05-16 01:27:48

@Hoon Jo has joined the channel

jsturtevant

2021-05-25 17:23:06

fyi on some potential release changes in packer goss provisioner @codenrhoden

caulagi (https://github.com/caulagi)

#53 packer 1.7 requires a sha256 checksum

I get an error like below with packer 1.7.2 ``` could not get sha256 checksum file for <a href="http://github.com/caulagi/goss">github.com/caulagi/goss</a> version 3.0.3. Is the file present on the release and correctly named ? GET <a href="https://github.com/caulagi/packer-plugin-goss/releases/download/v3.0.3/packer-plugin-goss_v3.0.3_SHA256SUMS">https://github.com/caulagi/packer-plugin-goss/releases/download/v3.0.3/packer-plugin-goss_v3.0.3_SHA256SUMS</a>: 404 [] ``<code> Also,</code>packer init<code>expects a sha256 checksum file of form **goss_v3.0.5_SHA256SUMS. So I changed the workflow to match what is used in</code>packer-plugin-scaffolding`

Comments

codenrhoden

2021-05-25 17:24:31

interesting catch there! Thanks for commenting on that.

K8sVoodoo

2021-05-25 18:09:32

@K8sVoodoo has joined the channel

Thomas Jordan

2021-06-04 04:09:12

@Thomas Jordan has joined the channel

Kevin Breit

2021-06-04 15:28:23

@Kevin Breit has joined the channel

Kevin Breit

2021-06-04 15:28:33

I’m trying to build an OVA in image-builder from a CICD pipeline and it’s giving me an error saying python3 isn’t in the path. But here’s output showing that’s not necessarily true.

User's Python3 binary directory must be in $PATH
Location of package is:
Location: /root/.local/lib/python3.8/site-packages
$PATH is currently: /root/.local/lib/python3.8/site-packages:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Apricote

2021-06-08 10:31:47

@Apricote has joined the channel

roehlc

2021-06-11 14:00:27

@roehlc has joined the channel

Maximilian Rink

2021-06-15 11:56:37

hmm, what is the difference between raw and qemu build targets?
as far as i can see its only the disk image format. anything im missing here?

codenrhoden

2021-06-16 21:06:30

The expectation is that they will diverge. We'll see if that actually happens... I'm with you, though.

Maximilian Rink

2021-06-16 22:01:12

yeah, ive come across that building images fore bare metal and as ironic supports cqow ive left it at qemu for now. Saves my builder a lot of space 😄

Maximilian Rink

2021-06-16 22:02:30

in general, im going to put up a pr for both builders tomorrow, bringiong goss and the additional commands from opther builders to this builders as well.

Maximilian Rink

2021-06-16 22:03:12

they worked fine for me and i see no reason to diverge here 😄 correct me iof im wrong @dan @jdetiber

jdetiber

2021-06-17 00:18:58

sgtm

Maximilian Rink

2021-06-17 09:36:51

😄

MaxRink (https://github.com/MaxRink)

#634 add goss and custom provisioner to qemu and raw

What this PR does / why we need it: adds GOSS and arbitrary post comands as every other builder already has Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # Additional context Add any other context for the reviewers

Labels

cncf-cla: yes, needs-ok-to-test, size/M

Comments

William Lam

2021-06-15 13:46:26

Not sure if this is the right place to ask, but it looks like the documentation for building your own custom OVA images to use with TKG is out of date with latest v1.20.5 at the least the directory structure and instructions no longer match up

docs.vmware.com

Build and Use Custom OVA Images on vSphere

Original URL: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2/vmware-tanzu-kubernetes-grid-12/GUID-build-images-capv-images.html

vrabbi

2021-06-15 15:39:42

Have you tried with the updated docs for tkg 1.3?

vrabbi

2021-06-15 15:40:07

They completely changed the mechanics and how to do build an image

William Lam

2021-06-15 15:53:16

Ah, I did try subbing in 1.3 but maybe it should have been 1.3.1. Let me take a look at the docs again, shame it doesn’t have a drop down to jump to specific version

William Lam

2021-06-15 15:53:53

ah I see it now, thanks Scott!

vrabbi

2021-06-15 15:56:01

Its all in docker now and is much easier to use

vrabbi

2021-06-15 15:56:21

If you have any issues with it let me know. More than happy to help out

William Lam

2021-06-15 17:03:18

Yes, this is MUCH easier! I’m able to build a default image now

Do you happen to know where the packer definitions are within container image to specify which ISO to use?

vrabbi

2021-06-15 17:38:37

it should be in /home/imagebuilder/packer/ova/

vrabbi

2021-06-15 17:39:00

the files there are:
centos-7.json
esx.json
linux
OWNERS
packer-common.json
packer-haproxy.json
packer-node.json
packer-windows.json
photon-3.json
rhel-7.json
ubuntu-1804.json
ubuntu-2004.json
vmx.json
vsphere.json
windows
windows-2004.json
windows-2019.json

vrabbi

2021-06-15 17:39:17

which OS are you trying to change the image for?

codenrhoden

2021-06-16 21:07:56

@William Lam All the JSON files in the container image are just copied straight from the repo, so it has all the same default whether you are using a container or not. All of those variables can be overriden via flags or your additional custom JSON file so you don't have to go in and edit files within the container.

William Lam

2021-06-16 21:08:27

Yup, figured it out with some pointers from @vrabbi

:partyk8s: codenrhoden

M.C. Attaway Jr

2021-06-15 23:07:44

@M.C. Attaway Jr has joined the channel

whites11

2021-06-16 14:56:43

@whites11 has joined the channel

whites11

2021-06-16 14:58:40

Hello all 👋
I might be missing a point, can anybody rubberduck me with my attempts to build an azure SIG image?
I deployed a VM on azure, installed all the deps and I am trying to run make build-azure-sig-ubuntu-1804
I am getting this error

Error initializing core: error interpolating default value for 'crictl_url': template: root:1:66: executing "root" at crictl&#95;version...>: error calling user: test</pre>But the crictl_version var is defined in one of the packer files:<br><pre>$ grep "crictl&#95;version" packer/azure/**.json<br>packer/azure/packer.json:    "crictl&#95;url": "<a href='https://github.com/kubernetes-sigs/cri-tools/releases/download/v{{user'></a>crictl_version}}/crictl-v{{usercrictl_version`}}-linux-amd64.tar.gz",
packer/azure/packer.json:    "crictl_version": "1.21.0",

What am I missing? Thanks

codenrhoden

2021-06-16 21:10:00

@cecile @jsturtevant I know it's last minute, but I'm not going to be able to make office hours this week (tomorrow morning). Attendance has been pretty low, but I'm hesitant to just cancel it. My plan is just to write in the doc that I won't be there, but am hoping any discussion makes its way into notes. Does that seem reasonable?

cecile

2021-06-16 21:11:47

let’s see if anyone else has agenda items by EOD and if not we can just cancel

cecile

2021-06-16 21:12:05

I don’t have the host key btw, does anyone else have it?

codenrhoden

2021-06-16 22:08:57

I do have it, yeah. 🙂 That's something I could pass along to you

jsturtevant

2021-06-17 15:52:04

Doesn't look like we have an agenda. Should we cancel for today then?

👍 cecile

Maximilian Rink

2021-06-17 13:52:45

btw, what are your thought around systemd-boot? together with systemd248+ it is possible to get working discencryption with TPM autounlock that isnt a hassle to maintain. Im currently looking into that for our bare metal nodes

naadir

2021-06-17 13:54:56

we'd want official distro support i think

naadir

2021-06-17 13:55:25

using it where the distro doesn't support it might be problematic in terms of support from the vendor

Maximilian Rink

2021-06-17 14:24:49

arch it is then 🙂 😛

naadir

2021-06-17 14:26:00

yolo

Maximilian Rink

2021-06-17 14:26:56

but tbh, systemd-boot has prooven more stable in our env than grub2 and configuration is sooo much easier

naadir

2021-06-17 14:43:25

need to convince the debian folk, not me

naadir

2021-06-17 14:44:05

i went to Lennart's talk on systemd-boot a few years ago at Fosdem and was mostly convinced then

naadir

2021-06-17 15:07:09

that said, @Patrick Daigle, would it be possible to ask Canonical if they would support Ubuntu images which use systemd-boot as the bootloader? Getting full disk encryption for free would be a bonus

Patrick Daigle

2021-06-17 15:09:09

No promises, but I can try bringing this up in our ongoing conversations and touch points.

:thank_you: naadir, Maximilian Rink

Patrick Daigle

2021-06-17 15:07:50

@Patrick Daigle has joined the channel

Maximilian Rink

2021-06-21 15:30:29

hey folks, if im not missreading this we are assuming that open-vm-tools is a base package in our goss checks

and

that is messing up my bare metal images, that obviously dont have that tooling.
is there any other provider requireing open-vm-tools besides vmware?
If not im ammending my PR for qemu goss tests to remove that from defaults

Maximilian Rink

2021-06-23 21:37:48

so, ive opened up a few PRs for the QEMU and RAW builder @jdetiber @codenrhoden

#634 add goss and custom provisioner to qemu and raw

#638 add efi support to raw and qemu builder

jayunit100-pub-chnl-plz

2021-06-22 18:50:56

@cecile or @codenrhoden can you reopen ? Feel like its
better for someone who knows the image-builder stuff to quickly make that PR and merge as needed.

I dont mind doing it but, its an OWNERS file so probably is something that can be done w/ a quick handshake by the regular imagebuilder owners :)

jayunit100 (https://github.com/jayunit100)

#611 add owners file for windows

Labels

cncf-cla: yes, size/XS

Comments

codenrhoden

2021-06-22 19:59:56

yeah I'm happy to do it

🎉 jayunit100-pub-chnl-plz

codenrhoden

2021-06-23 16:37:32

Opened

codenrhoden (https://github.com/codenrhoden)

#637 Add OWNERS for Windows Ansible roles

What this PR does / why we need it: Adds an OWNERS file specifically for the Windows Ansible roles. Owners there would be <a href="https://github.com/jsturtevant">@jsturtevant</a> and <a href="https://github.com/perithompson">@perithompson</a>. Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # Additional context None /hold /assign <a href="https://github.com/jsturtevant">@jsturtevant</a> <a href="https://github.com/CecileRobertMichon">@CecileRobertMichon</a> <a href="https://github.com/perithompson">@perithompson</a> <a href="https://github.com/jayunit100">@jayunit100</a>

Assignees

jsturtevant, jayunit100, CecileRobertMichon, perithompson

Labels

approved, cncf-cla: yes, do-not-merge/hold, size/XS

❤️ jayunit100-pub-chnl-plz

trierra

2021-06-22 23:51:01

@trierra has joined the channel

Kevin Breit

2021-06-25 19:15:48

  {% if kubernetes_semver is version('v1.21.0', '>=') %}
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
  {% endif %}

Is there a reason the SystemdCgroup section requires 1.21?

voor

2021-06-26 19:36:44

Just became the default in 1.21 so it was easier than making sure the underlying machine image was configured accordingly

Kevin Breit

2021-06-26 19:54:46

Makes sense. I think we are fine without it under 1.20. We will be on 1.21 within a few months is my guess anyways. Thank you!!!

👍 voor

Maximilian Rink

2021-06-28 18:18:20

yes, but you /really/ need it for 1.21, as otherwise you wont be able to bootstrap a node with the kubeadm settings capi uses for 1.21

trierra

2021-06-25 20:14:28

Hello! I'm might be missing something in the build flow and it would be awesome if someone can point me to the right direction.

I run make quick-release in order to build my changes, but I don't see a kubelet image despite I have modified code there

~/g/s/k/kubernetes (px-translation-library) [1]> ls -la _output/release-images/amd64/
total 422624
drwxr-xr-x 2 oksana oksana      4096 Jun 25 12:00 .
drwxr-xr-x 3 oksana oksana      4096 Jun 25 12:00 ..
-rw------- 2 oksana oksana 126286848 Jun 25 12:00 kube-apiserver.tar
-rw------- 2 oksana oksana 120957952 Jun 25 12:00 kube-controller-manager.tar
-rw------- 2 oksana oksana 133050368 Jun 25 12:00 kube-proxy.tar
-rw------- 2 oksana oksana  52460544 Jun 25 12:00 kube-scheduler.tar
oksana@dev-onaumov ~/g/s/k/kubernetes (px-translation-library)>

It looks like the kubelet has been built according to logs, but I can't find an output that I could load into a docker container and deploy later

+++ [0625 11:55:37] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/kube-scheduler

Maximilian Rink

2021-06-28 18:19:57

uhm, the image-builder doesnt have quick-release target, have you maybe picked the wrong channel?

trierra

2021-06-28 20:28:05

yeah, looks like that..

Nicklas Sörensen

2021-06-29 10:00:20

@Nicklas Sörensen has joined the channel

Kevin Breit

2021-07-01 03:00:37

How do you go about making sure the images are up to date when in production? I’d assume I’d want to routinely rebuild the images with a unique name (include date most likely) and update my CAPI machines to use them. If so, is there a way to add a suffix to the tmeplate name?

jsturtevant

2021-07-02 18:45:34

Basically yes. In practice, I think it will be slightly different for each provider and they way they are distributed.

jsturtevant

2021-07-02 18:48:22

@jackfrancis has been experimenting with a way to keep nodes fresh in a long running cluster: . It is an interesting approach you might checkout for ideas

jackfrancis/kamino

Kubernetes node clone factory

Stars

6

Language

Python

Kevin Breit

2021-07-02 18:53:18

Interesting. I’ll need to look into this a little more.

Kevin Breit

2021-07-02 18:53:32

My environment right now is VMware.

voor

2021-07-02 21:00:51

There's a content library for updated images of you're using the built-in kubernetes

Kevin Breit

2021-07-02 22:11:19

With Kamino?

voor

2021-07-03 01:39:42

No with vSphere

Kevin Breit

2021-07-03 02:39:02

I’m using ClusterAPI. Do you know if those images work for CAPI?

Kevin Breit

2021-07-03 02:39:14

I’d assume not through

Bala

2021-07-01 06:45:41

@Bala has joined the channel

Maximilian Rink

2021-07-03 15:40:38

hmm, we are currently not cleaning up the netplan folder after we build images. Is there a specific reason for that or is it just oversight 😄

naadir

2021-07-05 10:55:37

oversight. i also noticed we're not cleaning up the journald log

Maximilian Rink

2021-07-05 15:33:09

hmm, is there an ansible module that can do that? from the man page --vacum** only cleans up archived data from journald

Maximilian Rink

2021-07-05 15:33:45

journalctl --rotate
journalctl --vacuum-time=1s

Maximilian Rink

2021-07-05 15:33:51

might do the trick tho

Maximilian Rink

2021-07-05 15:34:11

still feels dirty to use cmds 😄

naadir

2021-07-05 15:34:32

welcome to use python-dbus if you want

naadir

2021-07-05 15:36:11

though you've not lived until you've run COM inside .NET inside Ruby inside Java to interact with Windows logging.

Maximilian Rink

2021-07-05 15:38:43

uhm, my worst sinn actually is realtime programming in windows 😄
Running a Soft-PLC on an Intel Atom with one core to run a pharmaceutical machine is pain

Maximilian Rink

2021-07-05 15:38:58

for everyone involved

naadir

2021-07-05 15:41:18

😬

naadir

2021-07-05 15:42:02

anyway, Poettering said in 2015

The journal will not be bus enabled any time soon, as dbus-daemon logs to it, and this would hence mean a cyclic dependency where dbus daemon logs to journald, and journald uses dbus-daemon's IPC services... This can be fixed only when we have kdbus where the whole broken idea of userspace IPC is gone...

Maximilian Rink

2021-07-05 15:43:58

😄

MaxRink (https://github.com/MaxRink)

#645 cleanup netplan files, cleanup journald

What this PR does / why we need it: Removes netplan files generated at install, if present Cleans up journald as best as possible Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # Additional context Add any other context for the reviewers

Labels

cncf-cla: yes

cecile

2021-07-07 17:39:34

📣 Hi all, since the last couple of image-builder office hours have had very low attendance and no agenda topics we decided to try something new. We’ll add the agenda entry to the notes a week before the next meeting so folks can add topics throughout the week (just added the one for next week in Image Builder Office Hours - Google Docs) and if there are no topics added by the night before the meeting we’ll cancel the meeting occurrence. Let’s try this out a couple of times and see how it works out.

👍 cpanato, voor, codenrhoden

👀 cpanato

:thanks: codenrhoden

cecile

2021-07-07 17:39:49

cc @codenrhoden @jsturtevant

:ack: jsturtevant

cecile

2021-07-14 22:58:54

no agenda added, should we cancel?

cecile

2021-07-15 00:26:51

Cancelling tomorrow’s meeting due to no agenda items

👍 jsturtevant

Zachary Wachtel

2021-07-09 21:19:48

@Zachary Wachtel has joined the channel

Kaushambi Sharma

2021-07-11 10:15:00

@Kaushambi Sharma has joined the channel

Christopher Dziomba

2021-07-12 10:44:03

@Christopher Dziomba has joined the channel

tamal

2021-07-12 21:29:21

@tamal has joined the channel

lastcoolnameleft

2021-07-13 20:35:51

@lastcoolnameleft has joined the channel

Tanner Watson

2021-07-14 01:58:23

@Tanner Watson has joined the channel

Vignesh Ram S

2021-07-14 10:19:19

@Vignesh Ram S has joined the channel

Hoon Jo

2021-07-15 05:00:02

@Hoon Jo has left the channel

Maximilian Rink

2021-07-19 21:09:48

@codenrhoden @naadir the ova ci is broken as credentials got rolled

codenrhoden

2021-07-19 21:16:47

it's concerning that those could get rotated without me knowing about it... Unfortnately I can't really dig into this right now (and of course it's like 1 week after we got it all turned on and working), so I may have to just turn it off for now.

naadir

2021-07-19 21:18:32

argh, ok

codenrhoden

2021-07-19 22:33:35

I sure wish the /override command worked. 🙂 I can open a PR to make this OVA CI optional later tonight, but have to run out the door right now. kid's tee ball game

cpanato

2021-07-20 09:05:37

will do that

cpanato

2021-07-20 09:09:15

cpanato (https://github.com/cpanato)

#22953 image-builder: make OVA job optional

make OVA job optional, need to update the job secrets. slack 🧵 <a href="https://kubernetes.slack.com/archives/C01E0Q35A8J/p1626725388053700">https://kubernetes.slack.com/archives/C01E0Q35A8J/p1626725388053700</a> /assign <a href="https://github.com/CecileRobertMichon">@CecileRobertMichon</a> <a href="https://github.com/codenrhoden">@codenrhoden</a>

Assignees

codenrhoden, CecileRobertMichon

Labels

cncf-cla: yes

Vignesh Goutham

2021-07-19 22:59:25

Hello all, I’m trying to run goss validate on an image built by image-builder with overrides manually, and I seem to be missing something. Packer running goss at end of run seems to work fine, but when I try to run goss validate with all the variables filled in, it seems to fail with index of nil pointer error, which I’m pretty sure is my mistake somewhere. Appreciate any help here. I filled in the goss-vars.yaml as well.

root@vignesh-6wdxg:~/image-builder/images/capi/packer# sudo goss -g goss/goss.yaml --vars goss/goss-vars.yaml --vars-inline '{"ARCH":"amd64","OS":"Ubuntu","PROVIDER":"ova"}' validate
Error: could not read json data in goss/goss-command.yaml: template: test:61:24: executing "test" at : error calling index: index of nil pointer

Vignesh Goutham

2021-07-19 23:16:04

I made it work providing all the overrides through --vars-inline, but would be interesting to know why populating the goss-vars.yaml file won’t work.

Maximilian Rink

2021-07-20 08:08:11

https://github.com/containerd/containerd/releases/tag/v1.5.4
We should probably update contained 😅

Maximilian Rink

2021-07-20 14:47:05

MaxRink (https://github.com/MaxRink)

#659 bump containerd to 1.5.4

What this PR does / why we need it: <a href="https://github.com/containerd/containerd/releases/tag/v1.5.4">https://github.com/containerd/containerd/releases/tag/v1.5.4</a> Tl;dr: fixes a CVE Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # Additional context Add any other context for the reviewers

Labels

cncf-cla: yes, size/XS

👍 bavarianbidi, cecile

Maximilian Rink

2021-07-20 20:25:49

hmm, im also seeing some GOVC import errors on new OVAs created by image builder

Maximilian Rink

2021-07-20 20:26:12

[20-07-21 19:18:39] Uploading ubuntu-2004-2021-07-20T18-58z-kube-v1.20.9-disk1.vmdk... Error: Post "": unexpected EOF
govc: Post "": unexpected EOF

Maximilian Rink

2021-07-20 20:26:32

Manual imports through the UI are fine tho, so i dont know what is going on here

Maximilian Rink

2021-07-20 22:03:07

Hmm, this is so weird, it seems to be a ci-only thing.
from my local machine /everything/ works, altough im literally just tunneling all traffic through the machine that runs the CI jobs

Jayesh

2021-07-23 18:29:19

@Jayesh has joined the channel

cecile

2021-07-26 21:40:01

Hi all, added an agenda entry for this Thursday’s office hours in Image Builder Office Hours - Google Docs - please add your topics items before Wednesday EOD

cecile

2021-07-29 01:21:35

Cancelling tomorrow’s meeting due to no agenda items

:ack: codenrhoden

:thanks: codenrhoden

Kevin Breit

2021-07-27 22:13:33

I’m (probably) going to need to load an internal root certificate into my images for CAPV. Is this possible with Image Builder?

Maximilian Rink

2021-07-29 10:52:42

yes, impliment your role that does that and set the custom role var

Maximilian Rink

2021-07-29 10:54:37

would be great if i could get a few more eyes on and , especiually the latter as i needed to touch goss and ansible for a few providers 🙂

#634 add goss and custom provisioner to qemu and raw

Maximilian Rink

2021-08-12 14:34:07

anyone? :D

Pankaj

2021-08-04 05:44:21

@Pankaj has joined the channel

naadir

2021-08-10 15:04:46

i'm getting lots of DMs with folk wanting . tbh, i think it's a bit suspect but seems like loads of orgs deploy .local domains

randomvariable (https://github.com/randomvariable)

#671 Add option to enable .local lookups via DNS in Photon

What this PR does / why we need it: Lot of folk using Photon are still using .local domains in contravention of the RFCs. This adds an option to enable resolution of .local via DNS rather than MulticastDNS. As this is technically a leak, it's disabled by default. Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # Additional context Add any other context for the reviewers

Labels

size/S, cncf-cla: yes

Comments

naadir

2021-08-10 15:05:00

cc @codenrhoden

vrabbi

2021-08-12 15:38:47

This would be great! I know .local shouldnt be used but i bump into .local domains all the time and in k8s without this workaround its an absolute nightmare

naadir

2021-08-12 17:01:25

it's merged @vrabbi but you'll need to enable it manually as it's technically a "leak", it's in the docs.

vrabbi

2021-08-12 17:01:40

awesome!

vrabbi

2021-08-12 17:01:42

thanks

vrabbi

2021-08-12 17:02:25

building an image now to test it out in my .local lab (i created a dedicated lab to test out .local domain issues with K8s)

👍:skin_tone_4: naadir

ysung

2021-08-19 16:55:33

@ysung has joined the channel

Aarush Goyal

2021-08-24 17:59:42

@Aarush Goyal has joined the channel

Maximilian Rink

2021-08-25 09:28:56

@codenrhoden @naadir @jdetiber we were thinking about pushing some of our baremetal code for image building upstream

Maximilian Rink

2021-08-25 09:30:05

In addition to the packer based builder we also have debootstrap for installing ubuntu, which is significantly faster and produces slimmer images

✔️ Shailesh Pant

Maximilian Rink

2021-08-25 09:30:21

is there any interest to add that to image builder?

naadir

2021-08-25 10:32:19

pinging @Anusha Hegde , @Jamie Monserrate, @Dharmjit and @Shailesh Pant from our Edge team who might have some opinions

i've been interested in debootstrap and the rhel-derivative equivalent. I would start off with an issue describing what you want to do, and we'll need active maintainers etc...

Shailesh Pant

2021-08-25 10:51:50

Thanks @naadir for adding, This most definitely is interesting and like you mentioned an issue detailing the approach and proposal would be a great place to start collaborating 👍

Shailesh Pant

2021-08-25 10:32:44

@Shailesh Pant has joined the channel

Dharmjit

2021-08-25 12:52:31

@Dharmjit has joined the channel

Anusha Hegde

2021-08-25 12:54:15

@Anusha Hegde has joined the channel

cecile

2021-08-25 18:20:37

Does anyone have agenda items / topics for tomorrow’s office hours? Image Builder Office Hours - Google Docs

codenrhoden

2021-08-26 15:13:29

I'm not seeing anything. Going to put the 'cancelled' banner in there I guess. 🙂

robbie

2021-08-26 12:00:55

@robbie has joined the channel

Vignesh Goutham

2021-08-30 20:02:36

Hello all, does anyone know of a specific reason why we do not use the minimal images for ubuntu and use the full size server images? Are there any caveats if we switch to the minimal ones?

whites11

2021-08-31 16:14:37

Anyone with experience in azure SIG image building for windows? CI for my PR fails and I can't understand what I'm missing 🙏

whites11

2021-08-31 16:15:13

This is last error I'm getting:

Build 'sig-windows-2019' errored after 1 second 43 milliseconds: the Shared Gallery Image to which to publish the managed image version to does not exist in the resource group image-builder-e2e-ey01nu[0m

jsturtevant

2021-08-31 19:45:29

i've started to look into this

:thank_you: whites11

jsturtevant

2021-08-31 19:46:40

I am a bit confused as I don't know why it passed, I thought each image name needed to be unique

jsturtevant

2021-09-01 01:28:41

opened

Mounish S

2021-09-03 13:37:15

@Mounish S has joined the channel

Jamie Monserrate

2021-09-06 04:16:34

@Jamie Monserrate has joined the channel

codenrhoden

2021-09-09 14:46:55

No agenda items in the doc this morning, wrote a note saying today's office hours are canceled. I've got a few things that probably make sense to add for next go around in 2 weeks.

👍 whites11, cecile

yoctozepto (Radosław Piliszek)

2021-09-13 10:26:29

@yoctozepto (Radosław Piliszek) has joined the channel

Vignesh Goutham

2021-09-16 00:46:53

Hello all, Im trying to do a qemu image build of ubuntu 2004 and I’m getting stuck on waiting for ssh to become available Any pointers here would be helpful.

==> qemu: Waiting for SSH to become available...
2021/09/15 23:44:29 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:2753...
2021/09/15 23:44:29 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2021/09/15 23:44:29 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2021/09/15 23:45:03 packer-builder-qemu plugin: [DEBUG] SSH handshake err: ssh: handshake failed: read tcp 127.0.0.1:51640->127.0.0.1:2753: read: connection reset by peer

jdetiber

2021-09-17 21:32:34

Ubuntu 18.04 CAPI image builds are currently broken for ova, qemu, and raw images, I created a PR to bump the version of the ISO we are using to fix:

Domingos Rodrigues

2021-09-21 01:21:14

@Domingos Rodrigues has joined the channel

Jeremi Piotrowski

2021-09-23 14:14:52

@Jeremi Piotrowski has joined the channel

Jeremi Piotrowski

2021-09-23 14:16:31

hi, I submitted the following PR to improve Flatcar support in image-builder and would love to get some feedback/reviews ; we're currently trying to bring up vsphere support as well

dongsupark

2021-09-24 15:02:26

Reviewed, thanks!

Jeremi Piotrowski

2021-09-24 15:45:40

thanks, i made adjustments and responded to your comments

Amim Knabben

2021-09-26 01:26:40

@Amim Knabben has joined the channel

farhan

2021-09-28 13:42:48

@farhan has joined the channel

Kevin Breit

2021-09-29 14:51:04

I need to install some trusted CA certificates into images for CAPI. These need to be for image repositories so it goes in directories specific to that repository address. Is there a built in task I can use for this or do I need to create a custom role?

vrabbi

2021-09-29 16:02:23

Why do this inside the image? That seems to me like a bad idea. If you need to replace the certs its a pain. I find it much easier to use files directives in kubeadmconfigtemplate and in kubeadmcontrolplane objects. That also allows for an easy rolling update with a changed cert without needing to change the template

Kevin Breit

2021-09-29 16:07:24

Interesting. Do you have an example of those pieces?

Kevin Breit

2021-09-29 16:11:50

Ahhhh just the content?

Kevin Breit

2021-09-29 16:41:49

What’s the difference between putting it in kubeadmconfigtemplate and kubeadmcontrolplane? Is it one is focused on control plane nodes and the other is worker nodes?

vrabbi

2021-09-29 17:07:18

Yep

vrabbi

2021-09-29 17:07:22

Exactly

vrabbi

2021-09-29 17:08:54

I use YTT for templating but here is an example of how i do it

vrabbi

2021-09-29 17:08:57

https://github.com/vrabbi/tkgm-customizations/blob/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/custom-ca/trust-harbor-cert.yaml

Kevin Breit

2021-09-29 18:10:39

And is this enough to get containerd to connect to an image repository whose certificate is signed by this CA?

Kevin Breit

2021-09-29 18:14:34

I’m trying to get the magic incarnation to make this work. I had it working last night and can’t reproduce it.

vrabbi

2021-09-29 18:54:15

If the ceet is in the OSes trusted CA certs then containerd should respect that

vrabbi

2021-09-29 18:54:21

At least it works for me

Kevin Breit

2021-09-29 18:54:54

PEM format too, right? Not DER or base64?

vrabbi

2021-09-29 18:55:17

Either base64 or PEM

vrabbi

2021-09-29 18:55:35

Which capi provider are you using?

vrabbi

2021-09-29 18:55:56

Look at the file directive in CAPV for the kubevip static pod

vrabbi

2021-09-29 18:56:12

You can see there how pem encoded would be passed

vrabbi

2021-09-29 18:56:25

It can alsi be b64 if you prefer

vrabbi

2021-09-29 18:56:46

I would not wish DER on my worst enimies so i dont know

Kevin Breit

2021-09-29 18:59:58

CAPV

Kevin Breit

2021-09-29 19:00:26

I’m new to PKI and all this so the format stuff is a bit of black magic. Is there a way to see which format something is encoded?

vrabbi

2021-09-29 19:01:08

Im not to farmiliar with that either but im sure that tools exist

vrabbi

2021-09-29 19:01:13

I just dont know

Kevin Breit

2021-09-29 19:18:22

Why are you installing it in the /etc directory structure then moving it later? Is it support for Proton?

vrabbi

2021-09-29 19:28:40

Photon and Ubuntu so need to support both

Kevin Breit

2021-09-29 19:50:21

I may have it working with your configuration. I’m not sure why it didn’t work when I was doing it manually but it’s possible I “tied a knot” earlier in the process that prevented it from working.

Kevin Breit

2021-09-30 04:21:42

https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=bb199da9-c629-46a4-b810-7dc4b2216e59

community.arubanetworks.com

Airheads Community

Product and Software: This article applies to all Aruba controllers and Aruba OS 3.1.1 and later. Question I have a certificate to upload to the Aruba

Original URL: https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=bb199da9-c629-46a4-b810-7dc4b2216e59

Kevin Breit

2021-10-11 14:52:18

Are you using image-builder to create your Photon image?

vrabbi

2021-10-11 15:13:27

Yes. Havent built photon for a bit of time but yes

Kevin Breit

2021-10-11 15:29:17

Is the process the same for Ubuntu and Photon? I'm trying Photon and it's not injecting the kickstart file

vrabbi

2021-10-11 17:55:34

Its the same

Kevin Breit

2021-10-12 14:43:34

*Thread Reply:* None

image.png

Kevin Breit

2021-10-12 14:44:06

This is what I’m seeing on proton

Kevin Breit

2021-10-12 14:45:53

    - PACKER_VAR_FILES=/opt/image-builder/images/capi/config.json make build-node-ova-vsphere-photon-3

That’s my build line

vrabbi

2021-10-12 14:47:08

im away travelling for 2 weeks and dont have my environment readily available to take a look. if you dont figure it out by then i will try and reproduce and see if i can figure out why its failing for you

Kevin Breit

2021-10-12 14:49:02

Thanks. Enjoy your travels!!!

Kevin Breit

2021-11-05 23:44:53

Not sure if you’re back but I still haven’t figured out the Proton issue.

Amim Knabben

2021-09-29 19:26:48

besides the local builds, is there a rule/best practice to bring some of the targets using docker?

Amim Knabben

2021-09-29 19:27:40

I have been struggling with the make deps-ova setup, in a lot of cases the python/deps gets a mess

jdetiber

2021-09-29 19:32:49

I've seen some instances where our dependency scripts have tried to install things in non-ideal locations before, it doesn't hurt to run it containerized if that is the case with the deps-ova target

✔️ Amim Knabben

Abhishek Mitra

2021-10-01 07:37:57

@Abhishek Mitra has joined the channel

Abhishek Mitra

2021-10-01 07:38:21

can someone help with a basic question . i was able to deploy my own custom AMI with cluster-api image-builder . However that doesnt show up in "clusterawsadm ami list" . Let me know how i can use my own custom AMI while deploying a cluster on AWS using the cluster-api . i have been searching for pointers in the documentation but couldnt find any
am sure this is something basic

voor

2021-10-01 12:02:30

You need to set the AMI id in the AWS machine template

Abhishek Mitra

2021-10-01 16:52:42

thanks for the pointer, i see the spec has the following

spec:
  template:
    spec:
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
      instanceType: t3.large
      sshKeyName: #####

Abhishek Mitra

2021-10-01 16:53:09

so here we need to add "image": "amiid" OR "image":"NAME"

Abhishek Mitra

2021-10-01 16:53:21

like capa-ami-centos-7-1.20.10-00-1631556951

Abhishek Mitra

2021-10-01 20:41:32

nm, figured it out . thanks !

voor

2021-10-02 13:07:51

Apologies, for folks that are finding this and haven't figured it out:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSMachineTemplate
metadata:
...
spec:
  template:
    spec:
      ...
      ami:
        id: ami-012345678910

Apricote

2021-10-04 12:23:07

Regarding the CAPI image: make build-qemu-ubuntu-2004 currently fails for me because no matching host key type found (full error message in a reply).

I think this is related to the latest OpenSSH release 8.8 () which disabled ssh-rsa with sha1 keys.
When I set the options mentionend in the OpenSSH release notes (ANSIBLESSHARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa" make [...]) it works again.

Should I open a ticket for this? I think we have to options here:

a) Update the Key (images/capi/cloudinit/{idrsa.capi.pub,user-data})
b) Add these options to the default ANSIBLESSH_ARGS

openssh.com

OpenSSH: Release Notes

OpenSSH release notes

Original URL: https://www.openssh.com/releasenotes.html

Apricote

2021-10-04 12:27:14

Error with context:

==> qemu: Executing Ansible: ansible-playbook -e packer_build_name="qemu" -e packer_builder_type=qemu -e packer_http_addr=10.0.2.2:8482 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=591e4e087ea2f5007e6c64deb382df58d419b7b6922eab45a1923d843d57615f pause_image=k8s.gcr.io/pause:3.4.1 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.5.4 crictl_url= crictl_sha256=44d5f550ef3f41f9b53155906e0229ffdbee4b19452b4df540265e29572b899c crictl_source_type=pkg custom_role= custom_role_names= disable_public_repos=false extra_debs= extra_repos= extra_rpms= http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_http_source= kubernetes_container_registry=k8s.gcr.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key=" " kubernetes_rpm_gpg_check=True kubernetes_deb_repo=" kubernetes-xenial" kubernetes_deb_gpg_key= kubernetes_cni_deb_version=0.8.7-00 kubernetes_cni_rpm_version=0.8.7-0 kubernetes_cni_semver=v0.8.7 kubernetes_cni_source_type=pkg kubernetes_semver=v1.22.2 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.22.2-00 kubernetes_rpm_version=1.20.9-0 no_proxy= python_path= redhat_epel_rpm= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= --extra-vars ansible_python_interpreter=/usr/bin/python3 -e ansible_ssh_private_key_file=/tmp/ansible-key801906084 -i /tmp/packer-provisioner-ansible4185103635 /home/jt/git/image-builder/images/capi/ansible/node.yml
    qemu:
    qemu: PLAY [all] *
==> qemu: failed to handshake
    qemu:
    qemu: TASK [Gathering Facts] 
    qemu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Unable to negotiate with 127.0.0.1 port 43119: no matching host key type found. Their offer: ssh-rsa", "unreachable": true}
    qemu:
    qemu: PLAY RECAP *
    qemu: default                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    qemu:
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
==> qemu: Deleting output directory...
Build 'qemu' errored after 5 minutes 18 seconds: Error executing Ansible: Non-zero exit status: exit status 4

==> Wait completed after 5 minutes 18 seconds

==> Some builds didn't complete successfully and had errors:
--> qemu: Error executing Ansible: Non-zero exit status: exit status 4

==> Builds finished but no artifacts were created.

Local OpenSSH Version:

$ ssh -V  
OpenSSH_8.8p1, OpenSSL 1.1.1l  24 Aug 2021

Kevin Breit

2021-10-04 14:26:41

@Apricote Is it breaking all builds or just certain configs

Apricote

2021-10-04 14:34:19

I only use the build-qemu-ubuntu-2004 target, so i can't tell if it is affecting other targets. Issue happens with latest master 5f3d1d6998a29ac1de63f0ae914bcb266a242078 for me.

Kevin Breit

2021-10-04 14:40:10

I may run a build today and see

Maximilian Rink

2021-10-04 21:42:00

@codenrhoden the ovas created by the vmware tool arent able to be imported into vcenter 6.5 via govc, only via UI

2021-10-04 22:39:57,899 |  ERROR: [04-10-21 22:39:56] Warning: Line 142: Unable to parse 'flags.vvtdEnabled' for attribute 'key' on element 'Config'.
[04-10-21 22:39:56] Warning: Line 143: Unable to parse 'flags.vbsEnabled' for attribute 'key' on element 'Config'.
govc: file does not exist

I suspect its failing because 6.5 cant make sense of the NVRAM config in the OVA, altough the OVA is still targeting 6.5
Do you know someone to pas that bugreport onto? 😄

kiran keshavamurthy

2021-10-12 18:13:16

What does the vmx-version in the OVF say?

Maximilian Rink

2021-10-12 18:13:31

👍 kiran keshavamurthy

Maximilian Rink

2021-10-12 18:15:37

The complete image building happens on vmx13, we basically upgrade the VM after deploy to the max the vcenter supports and set props like uuid for the CSI to function properly via govc

kiran keshavamurthy

2021-10-12 18:18:36

We saw something similar where OVA built using tar had issues getting imported into the vC from the UI. Hence ovftool was introduced as an option to build the OVA. Do you have access to ovftool to try that out? Either way, it seems to be an issue.

Maximilian Rink

2021-10-12 18:19:32

We already use the ovftool

Maximilian Rink

2021-10-12 18:19:47

At least I'm setting the env flag for it

Maximilian Rink

2021-10-12 18:20:00

The UI import is fine actually btw

Maximilian Rink

2021-10-12 18:20:07

That just works

Maximilian Rink

2021-10-12 18:20:19

Only via GOVC seems broken

kiran keshavamurthy

2021-10-12 18:20:29

ah sorry. I misread

richcase

2021-10-07 10:45:46

@richcase has joined the channel

2021-10-07 19:36:53

@PJ has joined the channel

Nadir Palacios

2021-10-08 02:44:11

@Nadir Palacios has joined the channel

Kevin Breit

2021-10-08 20:41:44

I’m trying to build a Proton image using a setup that already does Ubuntu. But Proton starts and says the network is unreachable and it can’t get the kickstart file.

TheKoguryo

2021-10-12 13:16:06

@TheKoguryo has joined the channel

kiran keshavamurthy

2021-10-12 19:51:19

@Maximilian Rink Regarding the cloud-init bug, Just wanted to check if you are working on an PR based on akutz’s suggestions here

Maximilian Rink

2021-10-12 19:59:11

No, I'm on leave this week

👍 kiran keshavamurthy

Gurpreet Singh

2021-10-14 18:15:09

@Gurpreet Singh has joined the channel

Kevin Latchford

2021-10-16 02:09:00

@Kevin Latchford has joined the channel

Shruthi Rajashekar

2021-10-18 06:05:57

@Shruthi Rajashekar has joined the channel

Shruthi Rajashekar

2021-10-18 06:26:00

Hello,
I am trying to install image builder project in my local environment and I have ansible installed

srajashekar@srajashekar-a01 capi % pip3 show ansible
Name: ansible
Version: 4.7.0
Summary: Radically simple IT automation
Home-page: 
Author: Ansible, Inc.
Author-email: info@ansible.com
License: GPLv3+
Location: /Users/srajashekar/Library/Python/3.9/lib/python/site-packages
Requires: ansible-core
Required-by:
srajashekar@srajashekar-a01 capi %

However, when I try to run make deps or make build-do-ubuntu-2004 to build an image, I am facing this error
make deps

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ansible 4.7.0 requires ansible-core<2.12,>=2.11.6, but you have ansible-core 2.11.5 which is incompatible.
Successfully installed ansible-core-2.11.5
User's Python3 binary directory must be in $PATH
Location of package is:
ansible
Location: /Users/srajashekar/Library/Python/3.9/lib/python/site-packages
$PATH is currently: /usr/local/opt/mysql-client/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/go/bin:/Users/srajashekar/bin::/Users/srajashekar/go/bin/ginkgo:/Users/srajashekar/Library/Python/3.9/lib/python/site-packages:/Users/srajashekar/wcp/image-builder/images/capi/.local/bin
make: * [deps-ami] Error 1

make build-do-ubuntu-2004

Requirement already satisfied: pycparser in /usr/local/lib/python3.9/site-packages (from cffi>=1.12->cryptography->ansible-core==2.11.5) (2.20)
User's Python3 binary directory must be in $PATH
Location of package is:
ansible
Location: /Users/srajashekar/Library/Python/3.9/lib/python/site-packages
$PATH is currently: /usr/local/opt/mysql-client/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/go/bin:/Users/srajashekar/bin::/Users/srajashekar/go/bin/ginkgo:/Users/srajashekar/Library/Python/3.9/lib/python/site-packages:/Users/srajashekar/wcp/image-builder/images/capi/.local/bin
make: * [deps-do] Error 1

voor

2021-10-18 10:32:47

Check out the container image for image builder, it might help with this issue.

Shruthi Rajashekar

2021-10-18 14:31:59

Can you please link me to it?

voor

2021-10-18 15:15:08

https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.9

GitHub

Release v0.1.9 · kubernetes-sigs/image-builder

This patch release includes bug fixes and cleanups, most notably an issue with containerd and SystemdCgroups when deploying K8s >= v1.21.0. The GA container image can be found at <a href="http://k8s.gcr.io/scl-ima">k8s.gcr.io/scl-ima</a>...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.9

:thank_you: Shruthi Rajashekar

Shruthi Rajashekar

2021-10-18 15:25:28

@voor when I try to run the make target for docker-build I’m getting this error

srajashekar@srajashekar-a01 capi % make docker-build
# We must pre-pull images 
docker pull docker/dockerfile:1.1experimental
1.1experimental: Pulling from docker/dockerfile
612615616619: Pull complete
Digest: sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa6412fff9caa6871649c44
Status: Downloaded newer image for docker/dockerfile:1.1-experimental
docker.io/docker/dockerfile:1.1-experimental
docker pull docker.io/library/ubuntu:focal
focal: Pulling from library/ubuntu
7b1a6ab2e44d: Pull complete
Digest: sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322
Status: Downloaded newer image for ubuntu:focal
docker.io/library/ubuntu:focal
bash: line 1: gcloud: command not found
DOCKER_BUILDKIT=1 docker build --build-arg PASSED_IB_VERSION=v0.1.10-66-gdab2b88f-dirty --build-arg ARCH=amd64 --build-arg BASE_IMAGE=docker.io/library/ubuntu:focal . -t gcr.io//cluster-node-image-builder-amd64:dev
invalid argument "gcr.io//cluster-node-image-builder-amd64:dev" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
make: ** [docker-build] Error 125
srajashekar@srajashekar-a01 capi %

voor

2021-10-18 15:28:21

You don't need to make the container it's already made at the repository url in the release notes

Shruthi Rajashekar

2021-10-18 15:33:08

I see so just pull the container and docker run?

voor

2021-10-18 15:33:52

Yes, you got it

:thank_you: Shruthi Rajashekar

Shruthi Rajashekar

2021-10-18 06:26:21

I installed ansible using pip3 install --user ansible

Shruthi Rajashekar

2021-10-18 06:26:49

I am new to image-builder and ansible and any help to set my env will be appreciated!

codenrhoden

2021-10-21 15:30:48

No Agenda items are present for today's office hours, so I marked the meeting as canceled.

Usama Ahmed

2021-10-22 07:40:03

@Usama Ahmed has joined the channel

Eric Graf

2021-10-29 12:56:15

@Eric Graf has joined the channel

Eric Graf

2021-10-29 13:10:50

Hi 👋, I opened a PR to add support for building OpenStack qemu-kvm CAPI images using a container. I read through the contributor guide but likely still missed something. Please let me know if there is something I need to change. Thanks!

Rohit

2021-11-02 16:21:40

@Rohit has joined the channel

nikparasyr

2021-11-02 16:22:23

@nikparasyr has joined the channel

sedefsavas

2021-11-03 14:05:26

@sedefsavas has joined the channel

sedefsavas

2021-11-03 14:06:35

Hello, will there be a release after cloud-init downgrade fix is in?

jimmi

2021-11-04 11:48:53

@jimmi has joined the channel

jimmi

2021-11-04 11:54:22

i'd love to see (set a default Containerd imports directory) merged and in a release soon. looks like all comments are addressed and the PR is ready to go. @jsturtevant @codenrhoden @Peri sorry to ping you directly but you've been helpful at reviewing the PR so far 🙏

GitHub

feat: set a default Containerd imports directory by dkoshkin · Pull Request #711 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adds a well-defined location to include additional Containerd configuration using imports. Wrapped it in an if statement to not break existing users who may be a...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/711

👀 codenrhoden

👍 jsturtevant

:git_merge: codenrhoden

codenrhoden

2021-11-04 14:34:09

No agenda items for today's office hours, so I marked it as canceled.

👍 cecile

jimmi

2021-11-04 15:19:46

thanks @codenrhoden @jsturtevant! very much appreciated

❤️ jsturtevant

jimmi

2021-11-04 15:52:14

out of interest, what is the process now for☝️to be used in provider image builds?

jsturtevant

2021-11-04 16:06:22

As in for a release of hte image-builder container?

jimmi

2021-11-04 16:08:17

sorry i'm a n00b in image-builder world... is there a doc or some reference i can read on the process? from what you said, i assume you mean there is a release of image-builder container and then individual providers can use it to build their respective images - is that right?

jsturtevant

2021-11-04 16:22:15

The docs are here: , For capz we usually build our reference images from the main branch but there is also container that has all the components build it in: . Looks like we might be due for a new release there.

jimmi

2021-11-04 16:47:09

thanks @jsturtevant

codenrhoden

2021-11-04 20:39:06

there's actually a 0.1.10 tag and container, I just never drafted the release notes.... But I agree, It's about time for a new one. There's a couple pending PRs that are pretty important, and once those are in I'll tag a new release.

👍 jsturtevant

cecile

2021-11-04 16:33:29

@codenrhoden for what would be the best way to test this? use any of the latest images built from main?

GitHub

Downgrade cloud-init. Workaround for #712 by kkeshavamurthy · Pull Request #728 · kubernetes-sigs/image-builder

What this PR does / why we need it: Fixes #722 Download 21.1-19 from launchpad and install it. Downgrade to 21.2-4.ph3 on photon-3 Latest available version on RHEL-7 is 19.4-7. Link. So, no change...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/728

codenrhoden

2021-11-04 17:44:35

@cecile that sounds right to me. Any recent images built with the Azure pipelines you have in place would have the latest cloud-init in them, and that cloud-init is buggy i a scenario triggered by CAPV. It doesn't appear to me that CAPZ images are effected, so we are leaving the cloud-init as-is. But yeah, if it turns out that CAPZ images are effected, you would know right away because the images wouldn't join into a cluster

cecile

2021-11-04 17:46:59

ok I can give that a try

cecile

2021-11-04 18:26:15

if tests pass we’re good

GitHub

Bump k8s version to the latest 1.22 patch version by CecileRobertMichon · Pull Request #1833 · kubernetes-sigs/cluster-api-provider-azure

What type of PR is this? What this PR does / why we need it: Bump k8s versions in tests to v1.22.3. Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number...

Original URL: https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/1833

:thanks: codenrhoden

Maximilian Rink

2021-11-06 17:36:11

@codenrhoden somewhat related: to we have end2end tests for actually standing up a CAPI cluster with the built images from CI on PRs to main?

Maximilian Rink

2021-11-06 17:38:32

If not: we have built something for Metal3 and CAPV internally, but on GitLab CI not on prow. We could port that over tho

codenrhoden

2021-11-08 18:26:27

so far it has been provider dependent. If I understand correctly, CAPZ does does test the images they they build with the Azure pipelines. I think CAPG does too with their nightly builds they've implemented. There is not ongoing E2E tests with up-to-date images for CAPV or CAPA that I am aware of, but I may be unaware. I would have to check with those projects.

Maximilian Rink

2021-11-08 13:21:46

Ubuntu 21.10 and 22.04 🧵

Maximilian Rink

2021-11-08 13:24:58

we need to rearchitect the image builder for newer ubuntu versions somewhat, as they only support subiquity. Ive started a while back with but there are a few more changes that we will need to make for newer ubuntu versions going forward. The question is, do we want to add ubuntu 2110 to image builder to start ironing out bugs before 2204 hits or do we want to wait?

GitHub

Add subiquity-based flavour for Ubuntu 20.04 based upon UEFI by MaxRink · Pull Request #408 · kubernetes-sigs/image-builder

As discussed in the office hours

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/408

codenrhoden

2021-11-08 18:24:43

that's a really good question. Maybe one for office hours? I do remember your PR re: subiquity, and I know it's gotten zero attention. 😞 It's definitely been a case of only looking at things for the LTS releases thus far.

I feel like we've mostly stuck to LTS releases thus far, but shaking things out ahead of time would be beneficial. I think the question becomes, do we remove a release, like 21.10, after 22.04 becomes available?

2021-11-11 17:56:25

Hi All 👋, I was going through the CAPI image builder code and noticed that base image for this dockerfile could use with an image bump (focal -> rolling). Would you all be open to a PR from me that does this bump?

cecile

2021-11-17 23:56:43

PR welcome. Just wondering, why rolling vs. latest? Isn’t “latest” latest LTS?

2021-11-18 00:22:41

I am okay with "latest" 🙂 I was assuming that, we would want to keep it to a specific release and update it as needed.
IIUC, this release specific tag, makes it possible potentially to create reproducible builds. But moving to "latest" will potentially mean the image tag is mutable and may result in different base image based on when the build was triggered for the same dockerfile.

cecile

2021-11-18 01:23:45

looking at Ubuntu - Official Image | Docker Hub it looked to me like rolling was even less release specific: “The ubuntu:latest tag points to the “latest LTS”, since that’s the version recommended for general use. The ubuntu:rolling tag points to the latest release (regardless of LTS status).”

hub.docker.com

Ubuntu - Official Image | Docker Hub

Ubuntu is a Debian-based Linux operating system based on free software.

Original URL: https://hub.docker.com/_/ubuntu/

2021-11-18 01:38:48

oh wow. I learnt something new today. Thank you @cecile. I will use latest in that case

👍 cecile

2021-11-22 18:45:43

PR created, waiting for jobs to pass:

GitHub

Switch base image from focal to latest by PushkarJ · Pull Request #740 · kubernetes-sigs/image-builder

What this PR does / why we need it: Current base image has many vulnerabilities that are fixed in newer release of ubuntu. Bumping this to latest will allow applying fixes to these vulnerabilities...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/740

cecile

2021-11-17 23:58:27

Hi all, are there any agenda items for tomorrow’s office hours? I don’t see anything on the meeting notes currently

codenrhoden

2021-11-18 15:17:24

I didn't see any as of this morning, so I just marked it as canceled. Let me know if you want to change that. I've got a couple things brewing that I'll hopefully put on the agenda in two weeks.

👍 cecile, jsturtevant

Ken Hamric

2021-11-22 16:20:59

@Ken Hamric has joined the channel

Shyam P R

2021-11-23 13:57:05

@Shyam P R has joined the channel

Shyam P R

2021-11-23 15:10:51

Team, in the default images that are created using the image builder project , what are the typical firewall/iptable rules in the images? I could not find any reference in the code. In our case, we are having to disable firewall as otherwise worker node to API Server communication, node to node communication, node port services etc do not work.

GitHub

GitHub - kubernetes-sigs/image-builder: Cross provider Kubernetes image building utility.

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder

voor

2021-11-23 19:47:42

Hi @Shyam P R most of those rules are not modified from whatever the upstream provided image is, so you can either modify them prior to image builder or afterwards.

Shyam P R

2021-11-24 03:03:32

Thanks @voor

Daniel Lipovetsky

2021-11-23 18:08:05

@Daniel Lipovetsky has joined the channel

Daniel Lipovetsky

2021-11-23 18:10:13

We recently merged to make it easier to override containerd configuration. However, it turns out that containerd only merges configuration at the section level (). Just wanted to raise awareness of this.

GitHub

feat: set a default Containerd imports directory by dkoshkin · Pull Request #711 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/711

GitHub

config merge via imports overwrites the wrong part of the config · Issue #5837 · containerd/containerd

Hi, I&#39;m trying to use imports to append a custom config to the containerd configuration as i want to avoid editing the config.toml files config.toml # Kubernetes doesn&#39;t use containerd rest...

Original URL: https://github.com/containerd/containerd/issues/5837

Francisco Navarro

2021-11-30 21:40:30

@Francisco Navarro has joined the channel

Amim Knabben

2021-12-01 12:44:35

is there a new 0.11 planned to be released?

Alejandro Cortina

2021-12-01 13:28:05

@Alejandro Cortina has joined the channel

mkumatag

2021-12-01 13:50:35

@mkumatag has joined the channel

mkumatag

2021-12-01 13:50:58

can someone help me reviewing this pr - ?

GitHub

set defaults for the ansible variables by mkumatag · Pull Request #739 · kubernetes-sigs/image-builder

What this PR does / why we need it: Set defaults for the variable used in the ansible roles, this will help the user to use the ansible independently without the packer tool. Made common_raw_rpms ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/739

:ack: codenrhoden

mkumatag

2021-12-06 09:59:21

@codenrhoden fixed all the review comments, ptal latest commit

codenrhoden

2021-12-02 15:11:18

No agenda items for office hours as of last night, marked it as canceled.
I will use that time to review the above PR, and the outstanding one for adding Rocky Linux to QEMU/Raw builders

codenrhoden

2021-12-02 16:51:13

I do think we should tag a v0.11. I think we are in a good spot for it, with some recent fixes going in for AMI+Amazon Linux 2, and some critical ones for OVAs (NTP fixes, Photon AppArmor).

Anyone know of anything pending that looks important? cc/ @Amim Knabben

Amim Knabben

2021-12-02 17:15:27

we have @Peri updates on a Windows nodes timezone issue + the capability to install openssh from a URL source

:ack: codenrhoden

whites11

2021-12-02 18:32:43

@whites11 has left the channel

Shyam P R

2021-12-07 04:30:51

Team, please review the PR to add Oracle Cloud Infrastructure(OCI) support in image-builder. Sorry this is my first PR, so apologies for any mistakes beforehand itself wrt to procedure/code.

GitHub

Add support for OCI Images by shyamradhakrishnan · Pull Request #754 · kubernetes-sigs/image-builder

What this PR does / why we need it: This PR adds support for Oracle Cloud Infrastructure(OCI) support in image-builder project. We are starting off with support to Ubuntu 18.04/20.04 and Oracle Lin...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/754

:ack: codenrhoden

Shyam P R

2021-12-13 16:13:15

Gentle reminder for this @codenrhoden @cecile @naadir

Shyam P R

2021-12-16 04:57:39

Gentle reminder for this @codenrhoden @cecile @naadir @kiran keshavamurthy

codenrhoden

2021-12-16 05:00:04

Ack. I know Kiran was able to take a look yesterday. I should have time tomorrow. Sorry for the delay.

Shyam P R

2021-12-16 05:14:49

np, Thanks @codenrhoden

Shyam P R

2022-01-07 14:57:09

Gentle reminder for this review @codenrhoden @cecile @naadir @kiran keshavamurthy

:ack: codenrhoden

:done_done: codenrhoden

Shyam P R

2022-01-13 14:22:47

Gentle reminder for the review @codenrhoden , the ova tests are still failing though, for all PRs

Shyam P R

2022-01-17 15:40:03

Gentle reminder for review @codenrhoden

Shyam P R

2022-01-20 15:50:33

Gentle reminder for review @codenrhoden

Shyam P R

2022-01-20 16:33:40

Thanks for the review @codenrhoden, @cecile it would be great to get your review also as your original approval was removed by an update

Shyam P R

2022-01-21 13:34:52

Thanks for the review @cecile

Kevin Breit

2021-12-09 21:59:38

Is there out of the box support for RHEL 8

Vasileios Mansolas

2021-12-10 00:14:28

@Vasileios Mansolas has joined the channel

Kevin

2021-12-10 21:21:51

@Kevin has joined the channel

mkumatag

2021-12-12 11:35:27

One more fix for the capi ansible code: , can someone help me reviewing this PR?

GitHub

Remove unwanted boot-order from the containerd service by mkumatag · Pull Request #758 · kubernetes-sigs/image-builder

What this PR does / why we need it:* cloud-final service already got the entries to start the containerd service as prereq here, and entries here in the containerd is not really needed. And there i...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/758

mkumatag

2021-12-12 11:35:40

cc @codenrhoden ^^

jsturtevant

2021-12-13 23:37:48

@codenrhoden It looks like OVA ci is failing consistently. It doesn't seem to be blocking test. Is this a known issue?

mkumatag

2021-12-14 06:41:00

Failing with this error:

Build 'vsphere' errored after 2 minutes 9 seconds: Post "": dial tcp 54.70.161.229:443: connect: connection timed out
199

200
==> Wait completed after 2 minutes 9 seconds
201

202
==> Some builds didn't complete successfully and had errors:
203
--> vsphere: Post "": dial tcp 54.70.161.229:443: connect: connection timed out

vrabbi

2021-12-14 07:45:44

I think it may have been blocked from internet access like many other vmc vcenters due to log4j cve. Not sure in this case but ive seen that in multiple vmc instances since the cve was announced

jsturtevant

2021-12-14 21:40:04

looks like this was the case

GitHub

fix ova-ci script to use vCenter IP directly by kkeshavamurthy · Pull Request #763 · kubernetes-sigs/image-builder

What this PR does / why we need it: Due to a CVE, VMC locked down certain access to vCenter on AWS. Adding the IP of the vCenter should force the connection to go thru the VPN and unblock the CI. I...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/763

codenrhoden

2021-12-16 04:51:06

sorry I didn't respond here, been a busy few days for me. I had Kiran get this fixed up once we got the right networking and firewall settings in place.

jsturtevant

2021-12-16 16:37:23

thanks!

codenrhoden

2021-12-16 04:55:01

I don't see any agenda items, so marked the office hours as canceled. I'll just throw out that I'd lake to tag v0.11 tomorrow. We've been talking about doing it for a while, and i was personally waiting until a recent Photon OVA issue was resolved, and it has been, so it seems like a good time. I know a few people have asked for it, and there's been a good number of Windows related fixes as well.

👍 cpanato, jsturtevant

jsturtevant

2021-12-16 20:47:30

can we include a docs publish? The windows docs aren't in the live page

codenrhoden

2021-12-16 21:21:20

that's... weird. let me check that out. docs are supposed to be published after every merge

codenrhoden

2021-12-16 21:40:10

I'll have to play with it a bit. It looks like it's not configured correctly. It's link to inside of capi.md, but in the published page it's a 404. I think I see why. It would also be nice to just have WIndows show up in the side-bar nav. It's all in the configuration I think, the docs are getting published fine, so it should be an easy fix.

👍 jsturtevant

jsturtevant

2021-12-16 22:25:20

so it is publishing the latest docs but just the page is missing from the nav?

jsturtevant

2021-12-16 22:25:30

let me know if there is anything can help with

codenrhoden

2021-12-17 21:50:30

@jsturtevant PR to get those Windows docs to show up:

GitHub

Link to Windows docs in TOC by codenrhoden · Pull Request #769 · kubernetes-sigs/image-builder

What this PR does / why we need it: mdBook doesn&#39;t generate/compile a doc page unless it is referenced from SUMMARY.md. Even though the windows page was linked to in capi.md, that wasn&#39;t en...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/769

❤️ jsturtevant

codenrhoden

2021-12-17 21:50:41

you'll be able to see a preview of the docs generated in the tests

codenrhoden

2021-12-17 21:51:25

The "deploy/netlify" job

jsturtevant

2021-12-17 23:23:05

Thanks!

codenrhoden

2021-12-16 21:22:24

tag v0.1.11 is made. And I finally published release notes for v0.1.10. homer-disappear
Will make sure v0.1.11 notes are done today.

:yay: jsturtevant

🎉 Kevin Breit, vrabbi, Apricote

codenrhoden

2021-12-17 04:53:36

GitHub

Release v0.1.11 · kubernetes-sigs/image-builder

This patch release includes new features, enhancements, security and bug fixes. The GA container image can be found at <a href="http://k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.11">k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.11</a> Deprecat...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.11

mkumatag

2021-12-17 06:38:26

Facing issues while make build-qemu-centos-7 , any idea what is happening?

image.png

mkumatag

2021-12-17 07:26:51

Adding following entry in the ansible.cfg helped me but its pretty slow

[defaults]
....
timeout = 120

codenrhoden

2021-12-17 22:08:06

I don't see anything obvious from the screen shot. Unfortunately I don't have any experience with the QEMU builder

mkumatag

2021-12-18 01:46:29

n/m, may be because of some nested virtulization, performance is not that great, will try running on a baremetal and check how this goes!

Amim Knabben

2022-01-06 15:10:15

I have one open PR that seems to be failing consistently on a not related job -

packer build -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/cni.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/containerd.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/common.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/additional_components.json"  -color=true -var-file="packer/ova/packer-common.json" -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/ova/photon-3.json" -var-file="packer/ova/vsphere.json" -except=esx -except=local -only=vsphere-clone -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/ci-photon-3.json"  packer/ova/packer-node.json
vsphere-clone: output will be in this color.
==> vsphere-clone: Cloning VM...
Build 'vsphere-clone' errored after 11 seconds 801 milliseconds: Error finding network: path 'sddc-cgw-network-8' resolves to multiple networks

GitHub

Adding virtualbox local build for local environment base box by knabben · Pull Request #699 · kubernetes-sigs/image-builder

What this PR does / why we need it: This PR adds a local build using VirtualBox hypervisor, packaged boxes are used directly on development environments like kubernetes-sigs/sig-windows-dev-tools. ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/699

Amim Knabben

2022-01-17 16:09:29

@codenrhoden Travis, can you PTAL again, cleaned up the open issues

GitHub

Adding virtualbox local build for local environment base box by knabben · Pull Request #699 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/699

codenrhoden

2022-01-06 17:34:09

@Amim Knabben (and anyone with PRs), we are definitely aware of the issues with OVA CI. I'm working to address the problem now (it's an infra issue). I'm hoping to have it resolved in the next day or so, but if it continues to be an issue we can consider making that CI test non-blocking. All OVA CI tests are going to fail right now.

Amim Knabben

2022-01-06 17:34:58

thanks Travis, lmk if you need help on it.

jsturtevant

2022-01-06 18:03:13

is there an issue/pr to track the fixes?

codenrhoden

2022-01-06 18:25:22

No not really. I can write an issue up, though, so it can be referenced. Will be a little while before there is a PR. I'll write one now.

codenrhoden

2022-01-06 18:29:36

GitHub

OVA CI consistently failing due to cluster network issues · Issue #775 · kubernetes-sigs/image-builder

What steps did you take and what happened: All OVA CI jobs are currently failing with the error: vsphere-clone: Error finding network: path &#39;sddc-cgw-network-8&#39; resolves to multiple network...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/775

:thanks: jsturtevant

jsturtevant

2022-01-06 18:43:51

I don't think OVA is marked as required right now...

codenrhoden

2022-01-06 18:49:35

wow, you are right. it was for a while, but I completely forgot that it wasn't anymore.

codenrhoden

2022-01-06 18:49:50

GitHub

test-infra/image-builder-ova-presubmits.yaml at master · kubernetes/test-infra

Test infrastructure for the Kubernetes project. Contribute to kubernetes/test-infra development by creating an account on GitHub.

Original URL: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/image-builder/image-builder-ova-presubmits.yaml#L10

codenrhoden

2022-01-06 18:50:41

I would have felt real silly if I went to go turn that on and found it was already there. Well, once this gets sorted out, I'll set it back. 😆

👍 jsturtevant

Shyam P R

2022-01-17 15:36:09

@codenrhoden the ova build seems to be failing, can we make it non blocking? sorry if it was already made that

codenrhoden

2022-01-19 18:50:08

They are currently non-blocking. And I will definitely get to this today.

:party_parrot: Amim Knabben

kuja53

2022-01-07 07:47:38

@kuja53 has joined the channel

cecile

2022-01-07 18:09:47

Hi all, I’m going to start a PR to update the OWNERs file and:

make sure various project owner aliases are up to date

remove image-builder approvers and reviewers that have not been active in the project in 1+ year

propose that we add @jsturtevant and @kiran keshavamurthy to image-builder-maintainers

If anyone has interest in becoming a reviewer, please let me know and we can work together to make it happen.

👍 timothysc, figo

❤️ jsturtevant, figo, codenrhoden, kiran keshavamurthy

cecile

2022-01-07 18:09:53

cc @codenrhoden

cecile

2022-01-07 18:10:53

let me know if there are any objections / considerations I should take into account before doing this. Also happy to split 3 from 1 and 2 if we want to approve the PRs separately.

cecile

2022-01-07 18:12:43

cc @akutz @jdetiber @figo @justinsb @luxas @moshloop @timothysc (you are all listed as maintainers currently)

timothysc

2022-01-07 18:13:51

feel free to rotate me out.

👍 cecile

akutz

2022-01-07 18:27:46

Ditto.

akutz

2022-01-07 18:28:26

If there is an emeritus tag, I'll take it since I created the project along with @jdetiber, but I am no longer actively involved.

❤️ cecile

cecile

2022-01-10 22:52:58

PR opened:

GitHub

Refresh OWNERs files by CecileRobertMichon · Pull Request #777 · kubernetes-sigs/image-builder

What this PR does / why we need it: makes sure various project owner aliases are up to date with their source project OWNERs ✅ moves inactive image-builder approvers and reviewers to emeritus (@ak...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/777

akutz

2022-01-07 18:12:55

@akutz has joined the channel

figo

2022-01-07 18:12:56

@figo has joined the channel

justinsb

2022-01-07 18:12:56

@justinsb has joined the channel

luxas

2022-01-07 18:12:56

@luxas has joined the channel

timothysc

2022-01-07 18:12:56

@timothysc has joined the channel

Yiyi Zhou

2022-01-11 19:51:52

@Yiyi Zhou has joined the channel

codenrhoden

2022-01-13 15:11:03

No agenda items for today's office hours. I marked it as canceled in the doc.

oguzdag

2022-01-14 11:09:45

@oguzdag has joined the channel

Yiyi Zhou

2022-01-18 21:49:21

hi, I am new to image-builder. This might be a stupid question. I was ruuning make build-node-ova-local-photon-3 and encountered FileNotFoundError: [Errno 2] No such file or directory: 'vmware-vdiskmanager' I googled this error and people saying VMware Fusion would already have vmware-vdiskmanager built in. Anyone has any ideas to fix this? Thank you!

codenrhoden

2022-01-19 18:48:41

Hi @Yiyi Zhou! It's most likely that the folder that contains vmware-vdiskmanager is not in your PATH environment variable. On my Mac, VMware Fusion adds that executable in this folder:

$ which vmware-vdiskmanager
/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager

codenrhoden

2022-01-19 18:49:04

so you probably need to add /Applications/VMware Fusion.app/Contents/Library to your PATH

Yiyi Zhou

2022-01-19 19:40:05

Thank you Travis! You are right. I added export PATH=$PATH:/Applications/VMware\ Fusion.app/Contents/Library and now it worked.

Yiyi Zhou

2022-01-19 20:22:19

@codenrhoden I have another question. So I didn't change any files and ran make build-node-ova-local-ubuntu-2004 . I opened the vmx file in Fusion, expecting to login with default username(builder) and password(builder) in packer-common.json. But it didn't work.

Yiyi Zhou

2022-01-19 20:50:19

I tried appending the key to the ~/.ssh/authorized_keys file locally, still unable to login.

GitHub

image-builder/user-data at 574bef90c6a73778c08ba0cd4fe5b5a3eae96543 · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/574bef90c6a73778c08ba0cd4fe5b5a3eae96543/images/capi/cloudinit/user-data#L262

codenrhoden

2022-01-20 17:19:38

Hi Yiyi. Since these images are intended for use with CAPI, they are setup to expect an interaction that CAPI automatically performs. Namely, this is the injection of cloud-init metadata to create a user and add an SSH key.

The best way to do about this when working with Fusion is to run the hack/image-post-create-config.sh script. The way I do it is to import the OVA into Fusion, but do not start the VM. Before you start the VM, run the above script, which will create the capv user and add the SSH key that is present in the image-builder repo. Then start the VM and you can SSH with something like ssh -i cloudinit/id_rsa.capi capv@IP.

I just took a look at the docs, and I'm noticing that is out of date. 😞 It definitely needs to be corrected

akutz

2022-01-20 17:24:20

FWIW Travis the issue was her local copy of the key lacked the correct file permissions, and SSH was balking at her.

akutz

2022-01-20 17:24:44

I pointed her to the hack/image-ssh.sh script, which should perhaps automatically apply the correct perms to the key in the examples area.

codenrhoden

2022-01-20 17:25:10

FYI, the builder user is definitely locked and can't be used. If you are booting the VMX directly (instead of importing the OVA first), you may be able to SSH into it by grabbing the IP from fusion, using the capv user, and pointing to the SSH key found in the cloudinit user, but I haven't done it that way in a while. But that set of files is injected with the metadata automatically here:

codenrhoden

2022-01-20 17:25:14

thanks @akutz

akutz

2022-01-20 17:26:07

And yeah, the builder user gets locked as part of the shutdown command.

GitHub

image-builder/packer-node.json at 574bef90c6a73778c08ba0cd4fe5b5a3eae96543 · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/574bef90c6a73778c08ba0cd4fe5b5a3eae96543/images/capi/packer/ova/packer-node.json#L16

Yiyi Zhou

2022-01-20 21:32:25

Thank you @codenrhoden @akutz

mkumatag

2022-01-24 11:36:23

@naadir can you please help me reviewing this PR? , I have tested the changes and works fine, cc @codenrhoden

mkumatag

2022-02-07 17:02:15

@naadir @codenrhoden can we merge if no other comments?

mkumatag

2022-02-15 11:29:40

Ping @naadir @codenrhoden

naadir

2022-02-15 14:45:55

I am not working on upstream at the moment. @codenrhoden @Sanika Gawhane, can we get this sorted please?

mkumatag

2022-02-17 10:19:50

@codenrhoden @Sanika Gawhane let me know if any meeting required to discuss this?

jsturtevant

2022-01-25 17:11:10

There is a new issue with WS2022 build in the Azure VHD job. It is an OS packaging issue and I don't have a workaround as of now so will open a PR shortly to disable the job until it is resolved.

jsturtevant

2022-01-25 17:14:49

fyi @codenrhoden @kiran keshavamurthy

:ack: kiran keshavamurthy

codenrhoden

2022-01-25 17:15:29

Thanks @jsturtevant

codenrhoden

2022-01-25 19:47:12

@jsturtevant If you like, me or @kiran keshavamurthy could open the PR to set required: false just to unblock things.

👀 kiran keshavamurthy

jsturtevant

2022-01-25 19:57:00

sorry was in sig meetings but I don't think we should disable all the tests just ws 2022:

GitHub

disable WS 2022 from ci by jsturtevant · Pull Request #787 · kubernetes-sigs/image-builder

What this PR does / why we need it: Windows server 2022 is failing to install SSH which is block PRs. This is an issue with the install media on WS not with our scripts. Will re-enable once this i...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/787

👍 kiran keshavamurthy

codenrhoden

2022-01-25 19:57:45

of course. picard_facepalm thanks!

jsturtevant

2022-01-25 20:41:20

Tests pass, ready for a review

codenrhoden

2022-01-26 19:28:42

OVA CI is working again. Finally!

❤️ jsturtevant, vrabbi, Amim Knabben, Vignesh Goutham

🙌 Shyam P R

:thank_you: cecile, vrabbi, Amim Knabben

codenrhoden

2022-01-27 15:01:16

No agenda items for this morning's office hours - marked it as canceled

neolit123

2022-02-01 11:51:06

📣 📣 📣 hello, a couple of message on behalf of the SIG Cluster Lifecycle leads
cc @cecile @codenrhoden (from the OWNERS file of image-builder)

annual reports: each subproject must nominate a person responsible to fill a short online form (it takes around 5-10 minutes, deadline Feb 11th):

please make sure that the details about your project (owners, contact, meetings) are up to date, if not you can PR changes:

more info on annual reports:

KubeCon EU maintainer track: if you are interested in doing a talk, reach out to the SIG leads and they can request a slot for you (deadline Feb 14th)

if you have not subscribed to the SIG mailing list, please do that!

thank you!

:done_ball: codenrhoden

cecile

2022-02-01 17:36:15

@codenrhoden @kiran keshavamurthy @jsturtevant should we jump on a quick call at some point and do the annual report together?

👍 jsturtevant, kiran keshavamurthy

cecile

2022-02-01 17:36:33

And anyone else who wants to participate

codenrhoden

2022-02-01 21:01:32

yeah, that would make sense to me. A lot easier with help!

cecile

2022-02-01 23:06:30

let me know when would be a good time for you

kiran keshavamurthy

2022-02-02 19:11:02

Maybe during the next office hours?

cecile

2022-02-03 00:02:25

I’m going to be oof end of next week, could we do this tomorrow or Friday?

jsturtevant

2022-02-03 16:15:45

today is pretty busy for me but Friday works

👍 cecile

codenrhoden

2022-02-04 16:49:49

I'm pretty free today as well. My group has been trying out "no meeting Fridays", so that works!

cecile

2022-02-04 18:12:51

I’m free too, no meeting Fridays FTW party_parrot

cecile

2022-02-04 18:13:07

wanna do this in 20 minutes?

codenrhoden

2022-02-04 18:31:41

ah, just saw this. I can do it! I didn't prepare though. 😬 @cecile

cecile

2022-02-04 18:32:20

no worries I did it for capz and capi already it’s pretty straightforward we can go through it together

cecile

2022-02-04 18:33:25

@jsturtevant @kiran keshavamurthy are y’all available?

👍 kiran keshavamurthy

jsturtevant

2022-02-04 18:35:22

i am

cecile

2022-02-04 18:37:25

filling out survey for SIG annual report :

Shyam P R

2022-02-02 15:14:43

Team, is there a way to skip installing open-vm-tools using image-builder? We are trying to use image builder in Oracle Linux 8 and in Arm architecture, and open-vm-tools is not present which is causing the build to fail.

Chaitanya

2022-02-07 08:44:47

@Chaitanya has joined the channel

Shyam P R

2022-02-09 12:17:15

Team, any reason we are not bumping the kubernetes version here - to something new? Also I dont see in customisation we have defined how to update kubernetes version - , is that something we should try to add?

GitHub

image-builder/kubernetes.json at master · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/master/images/capi/packer/config/kubernetes.json

Apricote

2022-02-09 12:41:11

From the book page that you linked:

The version of Kubernetes to install. The default version is kept at n-2.

As v1.23 is released, the default version should be updated to v1.21

Shyam P R

2022-02-09 14:08:07

Thanks @Apricote, looks like thes section si also misleading

See Customization section below for overriding this value

as the customisation section does not explain the kubernetes properties, If no on else picks it up, I will try to create a PR tomorrow

Shyam P R

2022-02-10 03:42:13

Can I please get an ok-to-test for the PR to bump up kubernetes version to n-2 and minor doc fixes?

GitHub

Bump up Kubernetes version to match the doucmentation(n-2), and minor… by shyamradhakrishnan · Pull Request #798 · kubernetes-sigs/image-builder

What this PR does / why we need it: Bump up Kubernetes version to match the doucmentation(n-2), and minor fixes in doc. The parameters to override Kubernetes version was not mentioned in the overri...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/798

codenrhoden

2022-02-10 15:40:22

No agenda items for today's office hours. Marked it as canceled. Will plan to take a look at the above PRs during that time.

Jake Plimack

2022-02-11 19:18:34

@Jake Plimack has joined the channel

codenrhoden

2022-02-24 15:33:56

Cancelled today's office hours - no agenda items

Mike Bars

2022-02-24 20:21:00

@Mike Bars has joined the channel

jlieb

2022-03-02 15:17:45

@jlieb has joined the channel

Vignesh Goutham

2022-03-02 19:08:46

Hello all, I’ve been using image-builder for a while now. I was exploring the cli for image-builder in the repo and It looks like it’s not under active development. Is this still on the roadmap? I can help contribute here if this is still a direction the community wants to pursue.

codenrhoden

2022-03-10 15:07:26

yeah, it's definitely true that the CLI isn't getting any attention at the moment. And has been for some time. 😞 There is no current agreed-upoon design of how to approach it, and it's been well over a year since it was even discussed.

For me and a team of engineers I work with, we've been in a state of "yeah, we will have resources to work on that in a month or two" for well over a year now. And honestly that is still the case -- something we still think we want to do, but just don't have the bandwidth/resources to look at it.

Vignesh Goutham

2022-03-10 23:32:35

ah, no worries. I will take a look at the repo and see if I can hash out a design when I get sometime. We can talk about it/discuss more in the office hours whenever I get some direction 🙂 Thanks for the info though.

Shyam P R

2022-03-08 04:46:48

Team, while doing make deps in an Ubuntu box, I get the following

Installing collected packages: wheel, pip
  WARNING: The script wheel is installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts pip, pip3 and pip3.8 are installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

but the documentation only talks about add the following in PATH

export PATH=$PWD/.bin:$PATH

Looks like both PWD and $HOME/.local/bin has to be on path. Is it OK if I create a PR to add both directories to path?

codenrhoden

2022-03-10 15:04:17

Seems like it would be fine to me. I wonder if this is some vestige of the docs being written when everything was on Python 2.7 and now things are fully moved over to Python 3.x

Shyam P R

2022-03-10 15:15:35

Thanks @codenrhoden I will create a PR for this

Shyam P R

2022-03-08 12:18:24

Can I please get an ok to test for the PR please

GitHub

Add support for Arm based images in OCI by shyamradhakrishnan · Pull Request #821 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/821

👍 jsturtevant

Shyam P R

2022-03-10 01:52:03

Team, please review the above PR

:done_done: codenrhoden

Shyam P R

2022-03-10 15:15:05

@codenrhoden I see you have done lgtm label but not approved for this, any reason? Sorry for bugging you

codenrhoden

2022-03-10 15:15:57

I try not to be the only person who reviews a PR

Shyam P R

2022-03-10 15:17:01

ooh ok, Thanks 🙂

codenrhoden

2022-03-10 15:17:29

sometimes it works out that way anyways, but I at least like to give some time for someone else to look

Shyam P R

2022-03-10 15:18:39

yeah that is great, more eyes the better, I am new to this 🙂, hence wanted to check.

Shyam P R

2022-03-11 11:33:35

Team, can we get another set of eyes on this PR please, we have go the review from @codenrhoden

jsturtevant

2022-03-11 18:22:50

i took a look, lgtm. on minor nit on comment

Shyam P R

2022-03-11 18:35:25

Thanks @jsturtevant, can I get an approve if thats ok from you or @codenrhoden, I will add the comment in the next Pr I have to raise to fix the thread -

Shyam P R (https://kubernetes.slack.com/team/U01MWTK8D6J)

Team, while doing make deps in an Ubuntu box, I get the following <code>Installing collected packages: wheel, pip WARNING: The script wheel is installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts pip, pip3 and pip3.8 are installed in '/home/ubuntu/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.</code> but the documentation only talks about add the following in PATH <code>export PATH=$PWD/.bin:$PATH</code> Looks like both PWD and $HOME/.local/bin has to be on path. Is it OK if I create a PR to add both directories to path?

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1646714808337879

:done_done: codenrhoden

Shyam P R

2022-03-15 13:24:55

Sorry to bother again for this @codenrhoden @jsturtevant but I had to rebase the PR for the ova ci job fix, updated the missing comment as well. Please review

Vignesh Goutham

2022-03-09 19:43:12

Hello team, does anyone have/know some sort of hardening that you use on top of the image-builder image? I found some generic OS hardening like dev-sec repo on GitHub, but I’m curious if something is standard across the k8s community.

codenrhoden

2022-03-10 15:02:48

I don't know of anything standard at the moment. I know of some vendors that use their own additional Ansible roles for hardening, but I don't know what the status of those becoming public is.

Vignesh Goutham

2022-03-10 23:30:41

got it, thanks. Do you mind sharing some links if you have them handy? If not, its fine 🙂

codenrhoden

2022-03-11 19:28:50

Everything I am aware of is non-public right now. I have good reason to believe that may change within a few months, but who knows when it comes to deadlines and releases. They always move around. 🙂

codenrhoden

2022-03-10 15:08:42

There were no items in the office-hours agenda, so I marked it as canceled for today.

Sayali Kulkarni

2022-03-11 07:58:35

@Sayali Kulkarni has joined the channel

mweibel

2022-03-11 09:06:36

@mweibel has joined the channel

Shyam P R

2022-03-14 08:05:04

Team, can someone with the vsphere image builder job knowledge check why the latest builds of pull-ova-all are failing(eg: ), it is failing for all the PRs, so don't think it is PR specific. The error is

[0;32m    vsphere-clone: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"echo /tmp/.ansible\"&& mkdir \"echo /tmp/.ansible/ansible-tmp-1647239786.000533-1078-173893619533102\" && echo ansible-tmp-1647239786.000533-1078-173893619533102=\"echo /tmp/.ansible/ansible-tmp-1647239786.000533-1078-173893619533102\" ), exited with result 1", "unreachable": true}[0m

TIA for the help.

Shyam P R

2022-03-15 16:27:49

Thanks @codenrhoden for fixing this

mweibel

2022-03-18 15:24:33

I randomly have errors with building custom windows 2019 AWS AMIs (using image-builder, pre pulling a few docker containers). It sometimes works and sometimes doesn't.
Issue seems to be the sysprep step in the end (executes C:/ProgramData/Amazon/EC2-Windows/Launch/Scripts/SysprepInstance.ps1). I'm looking into debugging this, but maybe anyone knows right away what that could be?

==> windows-2019: Provisioning with powershell script: packer/ami/scripts/sysprep_prerequisites.ps1
    windows-2019: Removing default unattend.xml file...
    windows-2019:
    windows-2019: TaskPath                                       TaskName                          State
    windows-2019: --------                                       --------                          -----
    windows-2019: \                                              Amazon Ec2 Launch - Instance I... Ready
    windows-2019: Using cloudbase-init unattend file for sysprep: C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\Unattend.xml
==> windows-2019: Provisioning with Powershell...
==> windows-2019: Provisioning with powershell script: /tmp/powershell-provisioner372200148
    windows-2019:
    windows-2019: C:\Users\Administrator>reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fDenyTSConnections /t REG_DWORD /d 1 /f
    windows-2019: The operation completed successfully.
==> windows-2019: Provisioning step had errors: Running the cleanup provisioner, if present...
==> windows-2019: Terminating the source AWS instance...

jsturtevant

2022-03-18 17:12:27

are there errors elsewhere in the output? I don't see specific

mweibel

2022-03-18 18:51:50

unfortunately not. There's a sysprep troubleshoot doc from AWS which might help, but requires connecting to the machine. Will try that on Monday.

docs.aws.amazon.com

Troubleshoot Sysprep - Amazon Elastic Compute Cloud

If you experience problems or receive error messages during image preparations, review the following logs. Log location varies depending on whether you are running EC2Config, EC2Launch v1, or EC2Launch v2 with Sysprep.

Original URL: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/sysprep-troubleshoot.html

jsturtevant

2022-03-18 21:45:57

troubling shooting sysprep is tough, sounds like you have a way to do it good luck!

mweibel

2022-03-22 09:08:48

thanks 🙂 funny enough I tried it several times yesterday and today and all of them worked. Fingers crossed that this continues 😉

Flask

2022-03-18 15:40:40

@Flask has joined the channel

Danny Brito

2022-03-21 23:12:50

@Danny Brito has joined the channel

dims

2022-03-24 12:46:15

@dims has joined the channel

dims

2022-03-24 12:46:46

@cpanato please see ( cc @codenrhoden and @cecile )

GitHub

Switch kubernetes_container_registry to registry.k8s.io by dims · Pull Request #844 · kubernetes-sigs/image-builder

Context: please see <https://groups.google.com/g/kubernetes-sig-testing/c/U7b_im9vRrM/m/7qywJeUTBQAJ> I am hoping this is the right spot to be able to switch all CAP* CI jobs to use the new registry ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/844

👀 cpanato

:ack: codenrhoden

dims

2022-03-24 12:46:53

@Arnaud (he/him) ^

Arnaud (he/him)

2022-03-24 12:48:39

Damn you’re too fast for me. I just opened a branch for it. 😅

:partycat: dims

cpanato

2022-03-24 12:57:00

nice

Arnaud (he/him)

2022-03-24 12:46:56

@Arnaud (he/him) has joined the channel

Batuhan Apaydın (developer-guy)

2022-03-24 13:49:31

@Batuhan Apaydın (developer-guy) has joined the channel

codenrhoden

2022-03-24 14:40:04

I see there are agenda items for this week's office hours. I've been on PTO this week, and won't be able to attend. I will be back online later today, though, so I'll look through notes and check out what was discussed.

👍 jlieb

cecile

2022-03-24 17:13:12

with return to office, the 8am meeting time doesn’t work well for me anymore as it’s right in the middle of my commute so I likely won’t be able to attend most weeks.

codenrhoden

2022-04-07 15:25:11

I think it's time to consider rescheduling this to a time that isn't at 8am for PDT folks. Our attendance is basically nil at this point. Though I am really appreciative of people leaving notes in the doc last week when no maintainers attended.

There are no agenda items this week, so I'm going to mark the meeting as canceled. Though there was an open through about what's up with the CLI somewhere, I'll track that down and respond to it here.

jlieb

2022-03-24 15:12:43

No maintainers attended @codenrhoden 🙂 There are some questions/requests to the maintainers in the notes. Enjoy your PTO!

mweibel

2022-03-29 06:09:21

👋 I have a few open PRs, could anyone review them, please? (mostly windows/AWS related):

👀 jsturtevant

jlieb

2022-03-31 15:19:12

Hi folks! I'm trying to get merged as part of our effort to ensure Flatcar images are aligned with the rest of the supported distros in terms of GOSS validations (right now some Flatcar builds are either broken due to GOSS errors or unvalidated due to "special" GOSS configuration which diverges from the generic one). This is a small PR (around 40 LOC changed) which ensures Flatcar has a similar GOSS config to the rest of the distros. The PR is blocking other work I'm doing in both image-builder and CAPV (because we need working vSphere images to test CAPV changes which we currently don't have).

@jsturtevant @moshloop looks like you've been automatically tagged as reviewers. Is there anything I can do to make it easier for you folks to review the PR? Alternatively, who could I tag for a review if both of you are busy these days? Thanks! 🙏

Xavier Avrillier

2022-04-07 13:44:52

@Xavier Avrillier has joined the channel

Xavier Avrillier

2022-04-07 13:45:12

@Xavier Avrillier has left the channel

codenrhoden

2022-04-07 15:30:29

@jlieb To answer your question about the CLI and it not compiling (which you fixed, thanks!), it is not currently in use. There were some plans around using it to do a few things, mainly as a pre-processor to generate the Packer configs (which would allow us to have much better logic ((and tests!)) around what the Packer configs look like. There are currently no design docs around it, and the work has no volunteers. The individual who was taking it up moved on to other responsibilities. So as you say, it hadn't been touched in quite some time.

I've been saying for at least a year that I thought I had some team members that may be able to pick up this work "in a couple months", but I've been saying that every few months now and it hasn't happened. Priorities... I wouldn't expect any effort on it in the short term, and if the effort does get picked up again, I think it would be starting from scratch to first get some agreement on goals and direction. I don't think the project has agreement on that, just a few personal opinions.

Vignesh Goutham

2022-04-07 20:25:59

I have also said few weeks back that I’ll take a look and propose some design, but didnt get time. I’m planning on writing something down next week, Hopefully I can get some thoughts and feedback from the community. If others have an idea or design, I’ll love to jump in and contribute as well.

jlieb

2022-04-08 13:55:01

Thanks a lot for clarifying Travis, and thanks for picking this up Vignesh! Just to clarify, there is nothing urgent on my end around this CLI - I was motivated to poke around and fix the build simply because my editor (VS Code) kept complaining about build failures while I was editing JSON files 😆 But of course it's nice to keep enhancing the project.

jlieb

2022-04-08 13:56:25

Is there a difference between the QEMU and the raw build targets in image-builder? I'm a bit confused by the existence of both since both targets seem to be using the qemu Packer builder type.

codenrhoden

2022-04-08 16:27:05

I'm not entirely sure if I'd say this has been the case or not (that the two have diverged. I don't really think they have from the Packer perspective, more from the Ansible perspective (there's a raw and a virt builder_type.

👍 jlieb

Ash

2022-04-11 23:15:39

@Ash has joined the channel

aniruddha

2022-04-12 16:51:26

@aniruddha has joined the channel

Dennis Lerch

2022-04-13 11:42:17

@Dennis Lerch has joined the channel

Amim Knabben

2022-04-19 14:16:01

@codenrhoden Hello, can you help review this one

GitHub

Modify permission of SSH folder by hxietkg · Pull Request #813 · kubernetes-sigs/image-builder

SSH is installed in folder 'C:\Program Files\OpenSSH', according to wiki from Win32-OpenSSH: <https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH> so we need the folder has correct...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/813

codenrhoden

2022-04-19 15:26:29

Took a look and /lgtm'd. I can approve if needed, too. Thanks for pinging me, as I had missed that folks were looking for my input.

Amim Knabben

2022-04-19 15:31:50

thanks Travis, would be great if you approve, we are in need of it to continue a few downstream testing

:done_done: codenrhoden

:ty: Amim Knabben

Abhay Krishna Arunachalam

2022-04-20 09:40:21

@Abhay Krishna Arunachalam has joined the channel

mkumatag

2022-04-20 15:56:05

Can someone help me reviewing this code - ? cc @codenrhoden

GitHub

Add PowerVS support by mkumatag · Pull Request #860 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/860

mkumatag

2022-04-29 18:18:06

@codenrhoden can you review or suggest someone who can review this PR?

codenrhoden

2022-05-05 15:17:17

Yeah, let me poke some people about it. I had thought this was related to Windows (my mind saw PowerShell), so was expecting windows reviewers to chime in. I was mistaken there.

👍 mkumatag

mkumatag

2022-05-05 15:24:59

n/m, this one is related to a different cloud platform for the ppc64le architecture.

mkumatag

2022-05-10 15:51:43

@codenrhoden wondering if you got a chance to poke anyone to review this PR?

codenrhoden

2022-05-10 15:53:17

Ack. Sorry, I have been swamped and distracted by sick kids lately. @kiran keshavamurthy, if you have any bandwidth, can you take a look at this one? It's on my plate as well.

mkumatag

2022-05-10 16:05:30

hope your kids are doing well now!

codenrhoden

2022-05-10 17:00:01

thanks. one of them is home from school with me, so lots of constant distractions!

mkumatag

2022-05-10 17:08:58

Ah.. happy parenting.. 🙂

mkumatag

2022-05-24 08:09:12

Gentle reminder - @kiran keshavamurthy I have fixed the review comment, can you please review the PR now?

👀 kiran keshavamurthy

mkumatag

2022-05-27 02:05:36

@kiran keshavamurthy ci is green now ptal.. @codenrhoden see if you can also ack..

mkumatag

2022-05-28 02:19:40

@codenrhoden I see Kiran lgtm'ed it, can you ptal the PR as well?

Abhay Krishna Arunachalam

2022-04-20 18:44:24

Hello all, I have a doubt: I'm curious why the open-vm-tools package installation is not a step in the Centos 8 OVA kickstart config. This causes Centos/RHEL 8 OVA builds to be stuck in Waiting for IP state, which then times out after 30 minutes. The VMWare console also shows that Open VMWare tools is not installed in this virtual machine, which corroborates the absence of the installation step. The installation is there in the Centos 7 ks.cfg. Is there a way to get around this?

Abhay Krishna Arunachalam

2022-04-20 18:56:57

cc @kiran keshavamurthy @codenrhoden

kiran keshavamurthy

2022-04-20 19:54:09

I think the centos-8/RHEL-8 support was added for qemu builders only and not OVA. So it has not been validated. We’d be happy to review a PR to add RHEL-8/Centos-8 support for OVAs.

Abhay Krishna Arunachalam

2022-04-20 20:02:09

That is correct. In our CI workflow, we added the following patch for it that adds the rhel-8.json to the OVA packer configs, and it uses the centos/http/8/ks.cfg
https://github.com/aws/eks-anywhere-build-tooling/blob/main/projects/kubernetes-si[…]ge-builder/patches/0009-Add-support-for-RHEL-8-OVA-builds.patch

Abhay Krishna Arunachalam

2022-04-20 20:04:24

could you let me know if this looks right?

Abhay Krishna Arunachalam

2022-04-20 21:36:23

GitHub

Add support for RHEL 8 Raw image builds by abhay-krishna · Pull Request #865 · kubernetes-sigs/image-builder

Adding support for RHEL 8 vsphere node OVA builds.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/865

Abhay Krishna Arunachalam

2022-04-20 21:37:31

☝️ this issue states

Without open-vm-tools installed, if you try to build centos/rhel OVA the process will remain stuck in "Waiting for IP address"

This is what we're seeing

Pearl

2022-04-21 09:46:48

@Pearl has joined the channel

Abhay Krishna Arunachalam

2022-04-22 21:00:33

When trying to build the Centos/RHEL 8 OVA, I get the error Kickstart file /run/install/ks.cfg is missing. I have tried serving the kickstart config checked in to the repo through both httpdirectory and floppydirs options with appropriate bootmediapath, but the build is not able to find the ks cfg file.

I also get the error Module floppy not found in directory /lib/modules/ . Is this because Centos 8 doesn;t support floppy modules or drivers?

Thilo Fromm

2022-04-26 15:52:27

@Thilo Fromm has joined the channel

Abhay Krishna Arunachalam

2022-04-26 19:34:31

@kiran keshavamurthy Can you take a look at this PR adding support for RHEL 8 OVA builds, including CI?

GitHub

Add support for RHEL 8 OVA builds by abhay-krishna · Pull Request #870 · kubernetes-sigs/image-builder

What this PR does / why we need it: This PR adds the necessary changes required to build RHEL 8 OVAs on vSphere. Tested on vSphere 6.7 with RHEL 8.4 DVD ISO. RHEL 8 removed floppy support so using...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/870

:ack: kiran keshavamurthy

Abhay Krishna Arunachalam

2022-04-27 18:12:20

Addressed comments here. Changed the structure of goss-vars a little to accommodate for different packages for different versions of the same OS. Could you take another look @kiran keshavamurthy

kiran keshavamurthy

2022-04-27 18:15:52

Thanks Abhay. I’ll try to look at it today or tomorrow.

Abhay Krishna Arunachalam

2022-04-27 18:17:51

Awesome, thanks, really appreciate it thanks2

Abhinav Sinha

2022-04-27 06:20:53

@Abhinav Sinha has joined the channel

Abhinav Sinha

2022-04-27 06:35:06

Hi folks! Can someone please confirm if there’s a way to set environment variables for, let’s say, directly setting the values of fields mentioned in images/capi/packer/config/kubernetes.json to build the images instead of using PACKERVARFILES? Or if there’s a workaround to achieve the same instead of having to manually edit the config file every time an image has to be built for a new k8s release?
Any help or clues would be really appreciated, thanks!

jsturtevant

2022-04-27 15:38:21

if you want to change just a few variables you can do something like PACKERFLAGS="-var=KUBERNETESVERSION=1.22.8 -var=ANOTHER_VAR=value" make build-target

:ty: Abhinav Sinha

Abhinav Sinha

2022-05-03 20:11:06

Thanks a lot @jsturtevant! That did the job 🎉

Amim Knabben

2022-05-02 19:00:27

Hey folks, got this issue on a MacOSX when building an OVA image, this happened on the windows-restart provisioner.

Has someone look into a similar issue using image-builder + macosx?

jsturtevant

2022-05-02 19:39:12

I haven't tried from mac os

codenrhoden

2022-05-05 15:16:00

And I haven't tried building Windows, so not sure about this one either. 😕

Kevin Breit

2022-05-05 18:46:19

I have a pipeline which builds a CAPI vSphere image using image-builder and it seems to be erroring but not giving details. Is there a way to look at why it’s happening or get logs from somewhere?

==> vsphere: Typing boot command...
==> vsphere: error typing a boot command (code, down) 28, false: ServerFaultCode: A general system error occurred: Invalid fault
==> vsphere: trying key input again
==> vsphere: Error running boot command: error typing a boot command (code, down) 28, false: ServerFaultCode: A general system error occurred: Invalid fault
==> vsphere: Clear boot order...
==> vsphere: Power off VM...
==> vsphere: Deleting Floppy image ...
==> vsphere: Destroying VM...
Build 'vsphere' errored after 48 seconds 532 milliseconds: Error running boot command: error typing a boot command (code, down) 28, false: ServerFaultCode: A general system error occurred: Invalid fault

jsturtevant

2022-05-06 18:50:45

looks related to packer, you can try turning on more verbose debugging:

packer.io

Packer by HashiCorp

Packer strives to be stable and bug-free, but issues inevitably arise where certain things may not work entirely correctly, or may not appear to work correctly.

Original URL: https://www.packer.io/docs/debugging

Kevin Breit

2022-05-06 19:03:40

I was able to get it working locally and not in the pipeline, which I thought was odd. But it works.

Amim Knabben

2022-05-10 18:07:26

hey folks, what is the safer way to add custom Ansible roles in the image-builder? Want to run a few tasks that are not in-tree

jsturtevant

2022-05-10 19:07:04

should be able to use customrole customrole_names?

:ty: Amim Knabben

Amim Knabben

2022-05-10 19:11:38

yea, I'm using volume mount from the host with custom ansible rules, just wondering other options

Amim Knabben

2022-05-11 14:20:34

- simple instructions in the customization

GitHub

Enabling custom roles instructions by knabben · Pull Request #886 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adding documentation on how to use custom roles on ansible Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will cl...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/886/files

jsturtevant

2022-05-11 16:45:14

awesome thanks!

Abhinav Sinha

2022-05-10 19:17:37

Hey folks! I’d like to know if there’s a way to customise the "ami_name" property in the packer config using the packer flag.

Yann Lev

2022-05-12 15:43:51

@Yann Lev has joined the channel

voor

2022-05-16 19:50:48

That feel when image-builder hangs waiting for the instance to come up, but the debug SSH key that it outputs works and connects to a running instance. feelsbadman

codenrhoden

2022-05-19 15:36:42

No agenda items for office hours today, so I marked it as canceled.
As a heads up, I plan to tag the repo today. There's a few things pending that will add some new capabilities, and it would be good to tag right before.

👍 jsturtevant, cecile

jlieb

2022-05-24 17:42:10

Hi folks! Would love to get some eyes on since there are multiple pending PRs blocked on it.
TL;DR: This PR commits the Ignition files used by Flatcar builds to the image-builder repo so that they don't have to be consumed from an external repo.

Thanks!

GitHub

Add Flatcar bootstrap Ignition files by johananl · Pull Request #895 · kubernetes-sigs/image-builder

What this PR does / why we need it: This PR adds a process for managing Ignition files used by Flatcar builds directly in the image-builder repo (until now these files used to be managed in an exte...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/895

Anirudh gopal

2022-05-24 22:16:52

@Anirudh gopal has joined the channel

Fredrik Björkman

2022-05-25 10:34:18

@Fredrik Björkman has joined the channel

Thomas Illingworth

2022-06-01 14:04:23

@Thomas Illingworth has joined the channel

codenrhoden

2022-06-02 15:45:59

no agenda items today, marking office hours as canceled (I was going to say I am not able to make it anways)
I know there are several open PRs that I've been pinged on. I'm blocking off some time this afternoon to play catch up on reviews.

tanisha banik

2022-06-06 19:38:20

@tanisha banik has joined the channel

Sriraman Srinivasan

2022-06-10 14:07:32

@Sriraman Srinivasan has joined the channel

aniruddha

2022-06-13 09:42:24

make build-gce-ubuntu-1804 giving me this error after I try to build an ubuntu 18.04 image

Jeremi Piotrowski

2022-06-13 09:50:58

Hi aniruddha, are you using a very recent ssh client version? Check this comment and apply the same fix to gce's ANSIBLESSHARGS value and see if that helps

GitHub

Fix build issues for Flatcar stable by kopiczko · Pull Request #907 · kubernetes-sigs/image-builder

This PR tries to fix various issues with Flatcar builds. This fixes multiple issues but at this point it isn&#39;t possible to test them in isolation. Fixes #906 Fixes #905 Workaround for #859 Also...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/907/files#r888283444

aniruddha

2022-06-13 10:32:45

@Jeremi Piotrowski Thanks 😄 it worked:handwithindexandmiddlefingerscrossed:

Marcus Noble (k8s@marcusnoble.co.uk)

2022-06-13 13:42:07

@Marcus Noble has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2022-06-13 13:43:22

Cross-post - (More details in thread)

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)

Does anyone know how I can go about debugging why <code>image-builder</code> times out waiting for SSH when trying to build CAPG images?

Original URL: https://kubernetes.slack.com/archives/C01D1RFEN9G/p1655122940063019

Jeremi Piotrowski

2022-06-13 13:57:52

This could be the exact same thing as just one message above yours (assuming CAPG -> GCP)

Jeremi Piotrowski (https://kubernetes.slack.com/team/U02E2PT10Q4)

Hi aniruddha, are you using a very recent ssh client version? Check this comment <a href="https://github.com/kubernetes-sigs/image-builder/pull/907/files#r888283444">https://github.com/kubernetes-sigs/image-builder/pull/907/files#r888283444</a> and apply the same fix to gce's ANSIBLESSHARGS value and see if that helps

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1655110258602719?thread_ts=1655109744.676069&cid=C01E0Q35A8J

Marcus Noble (k8s@marcusnoble.co.uk)

2022-06-13 14:04:51

I'm not sure it is. I did look at that but that doesn't seem to have any issues with SSH timing out (nor am I using the qemu variants)

Jeremi Piotrowski

2022-06-13 14:30:45

my bad, i thought i saw this resulting in ssh timeouts in the past as well (after multiple retries). it doesn't only affect qemu variants

kuja53

2022-06-13 14:45:25

I guess that best way is to connect to vnc endpoint which packer spawns on every run of image-builder. You could see password and port inside shell output.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-06-13 15:50:10

So, it looks like it was something related to my local network (not sure what exactly) but once I was on our corp VPN is worked first time. 🤷

Shyam P R

2022-06-14 05:41:05

Can I get please get couple of sponsors to join kubernetes-sigs org? I have contributed to image-builder and also planning to move our CAPI Provider repo for OCI to kubernetes-sigs org.

Shyam P R

2022-06-14 05:41:20

PRs

GitHub

Add support for Arm based images in OCI by shyamradhakrishnan · Pull Request #821 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/821

GitHub

Bump up Kubernetes version to match the doucmentation(n-2), and minor… by shyamradhakrishnan · Pull Request #798 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/798

GitHub

Add support for OCI Images by shyamradhakrishnan · Pull Request #754 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/754

cecile

2022-06-16 20:12:26

Happy to sponsor you

Shyam P R

2022-06-17 05:11:54

Thanks @cecile, i have got 2 sponsors, will ping you in case I need further help

Maximilian Rink

2022-06-20 17:20:11

hmm, ive ran into strange issues with ubuntu and qemu :S
Seems like the audit service isnt installed by default anymore on qemu systems. If i use the same iso and kickstart on an vCenter i still get the package

Maximilian Rink

2022-06-20 17:31:31

qemu: TASK [node : Ensure auditd is running and comes on at reboot] **
241 qemu: fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service auditd: host"}

Maximilian Rink

2022-06-20 18:22:52

looks like a regression from https://github.com/kubernetes-sigs/image-builder/commit/bc309118a3fe9db9c9d053e8d72d9fad7c43f1fa#diff-d74534cba8de8668a56[…]225ef8dae206955fffbf3135
as that also applies to the raw builder

GitHub

Adding support for CloudStack capi images · kubernetes-sigs/image-builder@bc30911

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/commit/bc309118a3fe9db9c9d053e8d72d9fad7c43f1fa#diff-d74534cba8de8668a566da82e02249667bf66b84225ef8dae206955fffbf3135

subhasmita

2022-06-25 06:12:53

@subhasmita has joined the channel

jlieb

2022-06-28 15:31:38

Hi folks. should be good to merge now. Testing is easy: make build-qemu-flatcar
I'd love to get this merged as soon as we can since there is a bunch of pending CAPI work that's blocked on this PR.
Thanks! 🙏

GitHub

Add Flatcar bootstrap Ignition files by johananl · Pull Request #895 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/895

MeghanaJangi

2022-06-29 11:03:41

@MeghanaJangi has joined the channel

Mohsen Vakilian

2022-06-30 19:58:40

@Mohsen Vakilian has joined the channel

Michal Mazurek

2022-07-02 14:58:58

@Michal Mazurek has joined the channel

Jeff Wu

2022-07-03 05:13:17

@Jeff Wu has joined the channel

jcepeda

2022-07-04 09:47:21

@jcepeda has joined the channel

swan

2022-07-05 10:17:25

@swan has joined the channel

swan

2022-07-05 10:19:11

Hi folks, I am trying to build photon OVA for vSphere, and trying to add a custom OVF property to the OVA, but defining the property in the json and setting that json file as env variable OVFCUSTOMPROPERTIES is not appending the desired property to OVF, could you please correct me if I am doing something wrong?

swan

2022-07-05 10:19:46

I am using target build-node-ova-vsphere-photon-3

swan

2022-07-05 15:37:05

I wanted to this property to be added to Cluster API Provider(CAPI) category

codenrhoden

2022-07-07 22:17:23

@kiran keshavamurthy should know how this works.

kiran keshavamurthy

2022-07-07 22:21:35

@swan are you also setting IB_OVFTOOL=1 to make sure to use ovftool to build the OVA.

swan

2022-07-08 10:03:30

Hey i have got this resolved, thanks @kiran keshavamurthy

Ness

2022-07-08 11:17:47

@Ness has joined the channel

faiq-archived

2022-07-12 17:56:29

@faiq-archived has joined the channel

Deepak Muley

2022-07-13 00:48:43

@Deepak Muley has joined the channel

Ankit Srivastava

2022-07-13 20:55:37

@Ankit Srivastava has joined the channel

Lucas Nascimento da Silva

2022-07-13 20:57:45

@Lucas Nascimento da Silva has joined the channel

opatrick

2022-07-19 03:37:56

@opatrick has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2022-07-19 10:26:49

Does anyone happen to know the minimum required IAM permissions needed by image-builder when building gcp GCP VM images? The docs say to give editor role but this is such large scope of unneeded permissions.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-07-19 12:57:03

I think it's just the following that's needed but would be great if anyone else can confirm.

compute.disks.create
compute.disks.delete
compute.disks.useReadOnly
compute.globalOperations.get
compute.images.create
compute.images.get
compute.images.getFromFamily
compute.images.getIamPolicy
compute.images.list
compute.instances.create
compute.instances.delete
compute.instances.get
compute.instances.getSerialPortOutput
compute.instances.setLabels
compute.instances.setMetadata
compute.instances.setServiceAccount
compute.machineTypes.get
compute.subnetworks.use
compute.subnetworks.useExternalIp
compute.zoneOperations.get
compute.zones.get
iam.serviceAccounts.actAs

Marcus Noble (k8s@marcusnoble.co.uk)

2022-07-20 15:59:44

If anyone is interested, I managed to confirm these permissions are enough today 🙂

👍 Ankit Srivastava, kiran keshavamurthy

Ondrej Kuchar

2022-07-19 17:10:31

@Ondrej Kuchar has joined the channel

Ankit Srivastava

2022-07-20 22:00:59

Building the image build-qemu-centos-7 takes around ~30 mins. Does anyone have any thoughts on how to make it faster? Is there a way we can have incremental image layers so that creating/testing images can be faster?

kiran keshavamurthy

2022-07-27 22:02:40

For vSphere OVAs, we build base images and re-use it (clone builder). This decreases the build times. Not sure if something similar exists for qemu

kopiczko

2022-07-21 10:17:15

@kopiczko has joined the channel

kopiczko

2022-07-21 10:19:20

is ready to merge for a while and solves issues with building images locally with newer OpenSSH versions. I'm not sure who I can assign for that. Can someone please have a look?

GitHub

Fix SSH/scp/SFTP issues for OpenSSH 9.0+ and Flatcar stable by kopiczko · Pull Request #907 · kubernetes-sigs/image-builder

This PR tries to fix various issues with Flatcar builds. This fixes multiple issues but at this point it isn't possible to test them in isolation. Fixes #905 Workaround for #859 Additional context ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/907

Jeremi Piotrowski

2022-07-25 10:37:00

hey @kopiczko, i've dropped a comment which i think would allow the CI to pass and allow this PR to get merged

mboersma

2022-08-01 22:08:25

@mboersma has joined the channel

Abhinay Singh

2022-08-09 09:36:50

@Abhinay Singh has joined the channel

Joe Kratzat

2022-08-09 15:51:57

@Joe Kratzat has joined the channel

mweibel

2022-08-12 09:12:26

my PR fails with FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip3.7' - anyone know why this could happen?

GitHub

feat: expose os_disk_size_db to allow overriding by mweibel · Pull Request #945 · kubernetes-sigs/image-builder

What this PR does / why we need it: Allows overriding VM os disk size.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/945#issuecomment-1212834678

jsturtevant

2022-08-12 19:24:20

that is a expected as its trying to isntall PIP I think, The vhds build and the 2019 image is failing with:

 Builds finished but no artifacts were created.
panic: runtime error: invalid memory address or nil pointer dereference
2022/08/12 07:50:00 packer-builder-azure-arm plugin: [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x1a933ce]
2022/08/12 07:50:00 packer-builder-azure-arm plugin: 
2022/08/12 07:50:00 packer-builder-azure-arm plugin: goroutine 226 [running]:
2022/08/12 07:50:00 packer-builder-azure-arm plugin: github.com/Azure/go-autorest/autorest/azure.(Future).WaitForCompletionRef(0xc0001f8cc0, 0x546d9c0, 0xc0006ba000, 0x53f6420, 0xc0000b98b0, 0x540e140, 0xc0001f4a50, 0xc000f08120, 0xc000cd2140, 0xdf8475800, ...)
2022/08/12 07:50:00 packer-builder-azure-arm plugin:   /home/circleci/project/packer/vendor/github.com/Azure/go-autorest/autorest/azure/async.go:174 +0x54e
2022/08/12 07:50:00 packer-builder-azure-arm plugin: github.com/hashicorp/packer/builder/azure/arm.(StepCaptureImage).captureImageFromVM(0xc000562e80, 0x546d9c0, 0xc0006ba000, 0xc0006e7b18, 0x18, 0xc0001a3120, 0x1c, 0xc0005626c0, 0xc000e0e3c0, 0x29)

jsturtevant

2022-08-12 19:24:48

click on "artifacts" on

jsturtevant

2022-08-12 19:25:27

at the bottom of the build log you can see:

jsturtevant

2022-08-12 19:25:28

sig-windows-2019: FAILED. See logs in the artifacts folder.
sig-centos-7-gen2: SUCCESS
sig-ubuntu-2004-gen2: SUCCESS
sig-windows-2022-containerd: SUCCESS
sig-centos-7: SUCCESS
sig-flatcar: SUCCESS
sig-ubuntu-1804: SUCCESS
sig-flatcar-gen2: SUCCESS
sig-ubuntu-1804-gen2: SUCCESS
sig-windows-2019-containerd: SUCCESS
sig-ubuntu-2004: SUCCESS

jsturtevant

2022-08-12 19:26:19

then you can naviagte to artifcats->azure-sigs and open sig-windows-2019.log

jsturtevant

2022-08-12 19:26:25

https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-bui[…]/945/pull-azure-sigs/1557989774642384896/artifacts/azure-sigs/

mweibel

2022-08-15 15:39:12

ah, thanks for the steps to debug - I didn't see it until now (and figured it out meanwhile myself) 😅

wondering what I could do about it - sounds like an issue with the azure packer plugin. Not sure if a retry would work?

jsturtevant

2022-08-15 17:15:05

looks like an issue in the scripts not naming the image properly. retrying the test for now is ok, looks like we got an issue open to look into it more

steve

2022-08-17 01:47:10

@steve has joined the channel

libsysguy

2022-08-18 16:41:45

@libsysguy has joined the channel

Neeraj Anand

2022-08-20 09:59:44

@Neeraj Anand has joined the channel

Abubakari Sadic

2022-08-21 20:49:37

@Abubakari Sadic has joined the channel

rahav jv

2022-08-23 10:54:27

@rahav jv has joined the channel

rahav jv

2022-08-23 10:54:48

Folks, wanted to check if there are plans to support photon 4.0

Marcus Noble (k8s@marcusnoble.co.uk)

2022-08-24 07:23:49

Morning y'all 👋 Looks like image-builder fails to build the new v1.25.0 release due to the change of default image registry (fails to pull coredns, likely more).
I've opened a PR to change the default registry to the new host:

GitHub

fix: updated image registry to new domain by AverageMarcus · Pull Request #956 · kubernetes-sigs/image-builder

Signed-off-by: Marcus Noble <a href="mailto:github@marcusnoble.co.uk">github@marcusnoble.co.uk</a> What this PR does / why we need it: Switched the default container image registry to the new <a href="http://registry.k8s.io">registry.k8s.io</a>. The new Kubernetes v1.25.0 release...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/956

swan

2022-08-24 08:55:48

I faced the same issue, we might need a release for the image-builder with this fix.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-08-24 09:00:43

Yeah. That'd be awesome.
I think it can be worked around by providing that registry value in an override vars file but having it work "out of the box" would be nice 🙂

swan

2022-08-24 09:02:25

oh right, i missed to see that override vars can accept registry key.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-08-24 09:03:24

I haven't tried it though. But pretty sure all those vars can be overridden with a provided --vars-file

swan

2022-08-24 09:03:45

I will give it a try

Marcus Noble (k8s@marcusnoble.co.uk)

2022-08-24 09:03:59

👍 Thanks 🙂

swan

2022-08-24 09:19:28

worked for me, the pull image error is gone, thanks @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2022-08-24 09:19:47

Nice! I'll add that as a note to the PR too 🙂

mimmus

2022-08-24 09:02:24

@mimmus has joined the channel

tariq

2022-08-26 21:11:52

@tariq has joined the channel

Shawn Wang

2022-08-28 07:22:11

@Shawn Wang has joined the channel

voor

2022-08-29 21:27:11

Is there a way to just skip the containerd steps in image builder?

voor

2022-08-29 21:28:06

I guess you could just comment out the

    - include_role:
        name: containerd

in node.yml

Amit Sahastrabuddhe

2022-08-30 06:10:29

@Amit Sahastrabuddhe has joined the channel

srm09

2022-08-30 20:13:35

@srm09 has joined the channel

srm09

2022-08-30 20:14:40

Hey folks, can someone chime in on whether the upstream CI is building 1.24 OVAs right now?

Clement Matundu

2022-08-31 19:21:08

@Clement Matundu has joined the channel

Phani Mvs

2022-09-01 06:33:42

@Phani Mvs has joined the channel

Kubernetes Moderator Service

2022-09-07 21:44:16

@Kubernetes Moderator Service has joined the channel

Kubernetes Moderator Service

2022-09-07 22:04:20

@Kubernetes Moderator Service has joined the channel

Sriraman Srinivasan

2022-09-08 13:37:39

I have create an issue for failing image builds (while installing iptables-persistent package). Also have submited PR for fixing the same.

GitHub

Update pyOpenSSL to latest by sriramandev · Pull Request #968 · kubernetes-sigs/image-builder

Signed-off-by: Sriraman S <a href="mailto:ssriraman@vmware.com">ssriraman@vmware.com</a> What this PR does / why we need it: This PR updates the pyOpenSSL package to the latest. This will fix the issue of iptables-persistent package instal...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/968

:thank_you: kiran keshavamurthy

Sriraman Srinivasan

2022-09-08 13:38:48

Just noticed, @jsturtevant has accepted for test... Thank you

jsturtevant

2022-09-08 16:26:43

Thanks for looking into it!

Cristiano Colangelo

2022-09-08 13:46:26

@Cristiano Colangelo has joined the channel

Vignesh Goutham

2022-09-12 19:13:08

Hi all, I’ve been observing a rather weird issue with vSphere image-builds lately in my environment. If the builder vm path I provide points a sub directory that is more than 1 level deep, rather than a top level directory - ex. capv/vignesh vs capv in the vsphere.json, the builder vm doesn’t seem to accept/take in the preseed cfg/kickstart file properly and loads into the GUI install prompt. Has anyone observed the same? Or any idea if there’s any folder level setting that I can take a look at? My vpshere is 7.0 and Im running the latest commit from image-builder.

Yannam C Chiranjeevi

2022-09-13 13:24:03

@Yannam C Chiranjeevi has joined the channel

Diego Braga

2022-09-14 16:34:43

@Diego Braga has joined the channel

Mitchel Haring

2022-09-15 23:33:04

@Mitchel Haring has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-16 06:16:57

The latest Kubernetes releases - 1.22.14, 1.25.1 - include a version of kubelet that now expects kubernetes-cni@v1.1.1 .
image-builder has it pinned to 0.8.7 causing (Ubuntu) builds to fail without overriding the kubernetescni* vars.
Is there a reason why we pin the version of kubernetes-cni package? Can we remove the version and instead rely on the dependency tree to install the needed version?
The vars could remain for those that need to pin to specific version but the default version can be set to * to allow ansible to install whichever version suits.

👀 swan

swan

2022-09-16 16:11:54

i am facing same issue, I even tried overriding parameters, but that didn’t work.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-16 21:55:14

I was able to get it to work by adding "kubernetescnidebversion": "**" to my PACKERVAR_FILES json to get it to build. Was that what you tried? Did that not work?

swan

2022-09-19 07:24:12

shall we set all the vars for kubernetescni** ?

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-19 11:49:08

For my test focusing on just ubuntu I only needed to overwrite the one var. ideally all should need updated to match the latest version available I guess.

swan

2022-09-19 12:26:14

I will give it a try, but eventually this has to be handled in image builder itself, isn’t it?

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-19 12:27:12

Yeah 100%. I'm just not sure how best to handle it as it could be a breaking change for some.

swan

2022-09-19 12:30:13

Would it help if we make a new release only with this change?

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-19 12:37:14

I would think that's enough, yes. Though I wasn't able to find out the backwards compatibility for the latest CNI. Not sure how many Kubernetes versions bank would still work as expected.

👍 swan

swan

2022-09-19 13:26:30

Also, I tried setting all CNI vars and it worked for Ubuntu, Flatcar and amazon OS, but for centOS its giving below error.

 Prevalidating AMI Name: capa-ami-centos-7-1.22.14-00-1663588776
==> centos-7: No AMI was found matching filters: {
==> centos-7:   Filters: [
==> centos-7:     {
==> centos-7:       Name: "root-device-type",
==> centos-7:       Values: ["ebs"]
==> centos-7:     },
==> centos-7:     {
==> centos-7:       Name: "virtualization-type",
==> centos-7:       Values: ["hvm"]
==> centos-7:     },
==> centos-7:     {
==> centos-7:       Name: "architecture",
==> centos-7:       Values: ["x86_64"]
==> centos-7:     },
==> centos-7:     {
==> centos-7:       Name: "name",
==> centos-7:       Values: ["CentOS Linux 7 x86_64 HVM EBS ENA**"]
==> centos-7:     }
==> centos-7:   ],
==> centos-7:   Owners: ["410186602215"]
==> centos-7: }

Anything changed for centOS too?

swan

2022-09-19 17:10:05

cc @sedefsavas

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-19 18:54:55

We don't work with CentOS images so couldn't say. 😞 Only have experience building Ubuntu and Flatcar.
Are you able to find that image if you search manually in the AWS console?

swan

2022-09-20 07:59:44

no that image is not there only for centOS and the recently released kubernetes versions

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 08:29:06

 Prevalidating AMI Name: capa-ami-centos-7-1.22.14-00-1663588776

Are you trying to build that image or make use of it? (Are you responsible for building the CAPA-provided AMIs?)

swan

2022-09-20 11:14:56

Yes you are right, I am responsible to build CAPA provided AMIs

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:16:05

That makes more sense 🙂
I assume the other Kube versions use the same CentOS base image. Did those also have issues finding the AMI?

swan

2022-09-20 11:16:26

yes other newer k8s releases also have same problem.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:17:05

Ah, maybe the upstream CentOS AMI name has changed? Let me see if I can find it myself.

:thank_you: swan

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:20:08

I can't seem to find anything from the 410186602215 account. Any idea what account that is? Doesn't look like it shares any AMIs

swan

2022-09-20 11:21:28

yeah even I couldnt find it, it is hardcoded in image builder

swan

2022-09-20 11:22:53

oh now I get it, I think you wont be able to find it, coz VMware uses internal AWS account to host these CAPA specific images

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:25:47

Yeah, looks like the CentOS ownerID to use for public images is 125523088429

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:26:32

But the names don't match at all.

swan

2022-09-20 11:27:10

I am wondering what might have changed, this worked for last k8s releases 🤔

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:27:17

If that's an internal VMWare account, wouldn't that mean no one else would be able to build a CentOS based image?

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:27:42

Maybe something in that account has changed? AMIs incorrectly deleted / set to private?

swan

2022-09-20 11:32:49

i checked the private AMIs, but didnt find there as well.

swan

2022-09-20 11:34:08

If it is account specific I will do some more digging, thanks for the help @Marcus Noble 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-20 11:34:46

No worries 🙂 If I get time I'll try and add CentOS to our pipeline to see if I can get it working or not but sounds like it might be an issue elsewhere.

🙇‍♀️ swan

swan

2022-09-29 16:23:20

I have found the issue for owner id mismatch, could someone take a look at this issue on priority as image building is blocked in CAPA for centOS distribution.

swan

2022-09-29 16:30:26

I can file a PR for this if no objection.
cc @kiran keshavamurthy @codenrhoden

kiran keshavamurthy

2022-09-29 19:54:36

@swan Feel free to open a PR

👍 swan

swan

2022-09-30 10:19:02

PR :
Is there any way I can inject this as env variable for generating AMIs, or do we need to wait for the release? If it’s latter, then we would be needing another release as soon as possible.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-30 10:21:54

You can provide the value as part of the —var-file

swan

2022-09-30 10:27:26

even ami specific filters? @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-30 10:29:21

Oh sorry I think you're right. That var isn’t exposed. 😞 yeah, release needed

:sadblob: swan

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-30 10:30:27

Thinking about it, might be worth updating that PR to use a user variable so it can be changed on the future

swan

2022-09-30 10:31:20

thats a good suggestion, we use these owner IDs as AMI filters in other OSes as well, I think there also we need the same update.

swan

2022-09-30 10:35:05

oh I see that amifilterowners is user variable

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-30 10:36:17

Oh nice! You should be able to override it then

swan

2022-09-30 10:36:31

yep will give it a try 🙂

swan

2022-09-30 10:39:00

worked 👍 thanks @Marcus Noble 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-30 10:40:08

Nice! Still worth getting a new release so it's fixed for everyone though.

swan

2022-09-30 10:40:42

Agreed!!! @codenrhoden @kiran keshavamurthy could a new release be taken care of?

swan

2022-09-30 12:03:40

we also need fix for cni versions used, image builder has hardcoded it to 0.8.x. A todo item before releasing new version.
Looks like its fixed in main branch

swan

2022-10-03 09:33:18

Bumping this thread again, could we get a newer release with newer changes as there are many fixes which could benefit the further image build for newer k8s releases

swan

2022-10-03 09:33:27

cc @codenrhoden @kiran keshavamurthy

kiran keshavamurthy

2022-10-03 17:56:40

I see 0.1.13 tag that @codenrhoden created recently.

swan

2022-10-04 06:30:39

I think that was created before this bug fix

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-04 07:13:50

The tag isn’t enough anyway. The container image also needs building and publishing.

:this: swan

swan

2022-10-06 08:57:17

@kiran keshavamurthy @codenrhoden @jsturtevant could you please help with the new release?

Kavitha Daula

2022-09-22 02:07:05

@Kavitha Daula has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-22 05:43:13

Looks like the 1.25.x builds are now failing due to the change of image registry

failed to pull image "k8s.gcr.io/coredns:v1.9.3"

Any chance we can get a new release? This fix is already in the main branch, just needs releasing 🙂

jsturtevant

2022-09-22 16:40:07

@kiran keshavamurthy are you able to help with releases? I haven't done one before

kiran keshavamurthy

2022-09-22 17:43:23

I’ve not as well.
Hello @codenrhoden, can you pls help us out here.

kiran keshavamurthy

2022-09-22 17:45:05

I believe Travis is the only person with the permission to cut a release. We should get a couple more folks added to that list. @jsturtevant if you are ok, I can open up a PR to get the 2 of added. Maybe @cecile too?

👍 jsturtevant

cecile

2022-09-22 17:47:40

I also haven’t done an image-builder release before, do we have docs on the process?

kiran keshavamurthy

2022-09-22 22:13:18

I don’t think so. Hopefully @codenrhoden is able to help out this time.

codenrhoden

2022-09-22 22:51:28

Hey folks. Definitely happy to help out however I can. I believe this is the location that would control being able to tag the repo: https://github.com/kubernetes/org/blob/44cf4faf10760dcc023dc4220b5e1a61875a93e1/config/kubernetes-sigs/sig-cluster-lifecycle/teams.yaml#L285

GitHub

org/teams.yaml at 44cf4faf10760dcc023dc4220b5e1a61875a93e1 · kubernetes/org

Meta configuration for Kubernetes Github Org. Contribute to kubernetes/org development by creating an account on GitHub.

Original URL: https://github.com/kubernetes/org/blob/44cf4faf10760dcc023dc4220b5e1a61875a93e1/config/kubernetes-sigs/sig-cluster-lifecycle/teams.yaml#L285

codenrhoden

2022-09-22 22:52:57

And if you need me to tag, I can do that too. Just confirm the right commit - I haven’t been looking at the repo lately (as I’m sure you’re aware) 😆.

kiran keshavamurthy

2022-09-22 23:04:34

Hey Travis 👋
Thanks a lot.
I think we can tag it with the latest commit 4b97ae8b85216ac9e5f187fe88a2097e7813e525 unless someone has any objections.

codenrhoden

2022-09-23 15:56:00

I pushed a tag for v0.1.13. That should kick off the staging container build as well. Hope it all works out!

:thank_you: Marcus Noble, kiran keshavamurthy, jsturtevant

kiran keshavamurthy

2022-09-23 19:05:26

Thanks Travis, Can you pls document the steps somewhere so that we have it for future.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-26 06:49:12

@codenrhoden it looks like a tag has been created but not a release, the latest is still v0.1.12. Looks like because of that the docker image hasn't been created either. Is there still something that needs to be done to get the release out?

:this: swan

kiran keshavamurthy

2022-10-03 17:57:05

@codenrhoden ping ^^

codenrhoden

2022-10-04 20:42:21

@kiran keshavamurthy next steps would be to test out the staging builds and if all looks good, promote the image to prod. Sanika definitely knows how to do all that. You can make sure that the staging build worked correctly in test grid.

Sorry, I definitely thought people were aware of those next steps. Regardless, I will write it up. Should I just drop it in an issue, add it to the official docs, or put it in the repo wiki (which I don’t think we use)?

:thank_you: kiran keshavamurthy, jsturtevant

:ack: Sanika Gawhane

Sanika Gawhane

2022-10-04 22:18:40

Hey @codenrhoden 👋
I was able to build AMI and Azure image with the staging container. Testing OVA build atm.
Here's the PR for promoting 0.1.13 image -

cc - @kiran keshavamurthy

GitHub

Promote v0.1.13 image by SanikaGawhane · Pull Request #4309 · kubernetes/k8s.io

Promote <a href="http://gcr.io/k8s-staging-scl-image-builder/cluster-node-image-builder-amd64:v0.1.13">gcr.io/k8s-staging-scl-image-builder/cluster-node-image-builder-amd64:v0.1.13</a> to prod. /cc @codenrhoden

Original URL: https://github.com/kubernetes/k8s.io/pull/4309

🎉 codenrhoden

codenrhoden

2022-10-05 00:10:14

Cool! looks like it is already approved. dancing-penguin2

:thanks: Sanika Gawhane

:yes: Sanika Gawhane

🎉 Marcus Noble

timmycarr

2022-09-26 20:32:55

@timmycarr has joined the channel

quiquell

2022-09-27 09:34:19

@quiquell has joined the channel

quiquell

2022-09-27 09:34:30

Hi I have put in place a PR to pass console to kernel params to debian based images , how do I regenerate the image ?

GitHub

debian kernel params console by qinqon · Pull Request #984 · kubernetes-sigs/image-builder

What this PR does / why we need it: When runnig a debian base image at kubevirt is not possible to run virtctl console, this change add the missing kernel params to allow that. Additional context D...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/984

Shyam P R

2022-09-27 14:03:50

you have to execute the make command again

quiquell

2022-09-29 08:11:23

Working now, thanks

Cristiano Colangelo

2022-09-27 14:01:45

what are the default credentials to access a node through ssh?

Shyam P R

2022-09-27 14:03:35

Can you explain a bit more? which node? The node created during image building process?

Cristiano Colangelo

2022-09-27 14:04:10

images build under images/capi/packer

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-27 14:04:15

It varies depending on which provider you're building for (some don't set at all). For AWS see

GitHub

image-builder/packer.json at ef9fed11d164c58dcb9e56699c56b0fe953f40d9 · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/ef9fed11d164c58dcb9e56699c56b0fe953f40d9/images/capi/packer/ami/packer.json#L45-L47

Cristiano Colangelo

2022-09-27 14:05:24

thanks

Shyam P R

2022-09-27 14:06:53

I dont think the ssh keys specified there are stored in the image so that when the an instance is created using the image, you can use the same ssh key. I may be wrong though. The ssh key is only valid for the instance which was used to create the image.

Cristiano Colangelo

2022-09-27 14:07:41

I tried to look in here (QEMU image) and I see some username and password but they don't work

GitHub

image-builder/packer.json at ef9fed11d164c58dcb9e56699c56b0fe953f40d9 · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/ef9fed11d164c58dcb9e56699c56b0fe953f40d9/images/capi/packer/qemu/packer.json

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-27 14:08:11

Oh, do you mean the image when finished and not during the building?

Cristiano Colangelo

2022-09-27 14:08:33

yes

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-27 14:09:00

I think you'd want to handle that during the userdata stage, not during the building. I might be wrong though.

Cristiano Colangelo

2022-09-27 14:10:26

basically I'm trying to debug cloud-init errors... masters are provisioned on the infrastructure but fail at some point during initialization

Cristiano Colangelo

2022-09-27 14:10:40

maybe some DNS problem, not sure

Marcus Noble (k8s@marcusnoble.co.uk)

2022-09-27 14:11:47

oh, that's not idea. I'm not sure then unfortunately. 😞

Cristiano Colangelo

2022-09-27 14:12:31

there should be a way though 🤔

Shyam P R

2022-09-27 14:13:11

which provider?

Cristiano Colangelo

2022-09-27 14:13:20

OpenStack

Cristiano Colangelo

2022-09-27 14:14:32

will try thanks

Ray Krueger

2022-09-29 19:14:12

@Ray Krueger has joined the channel

knfoo

2022-10-04 08:04:15

@knfoo has joined the channel

Drew Hudson-Viles

2022-10-05 17:47:08

@Drew Hudson-Viles has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-10 10:02:02

Is anyone using image-builder with ubuntu 22.04? Any issues? I'm looking to update if it's nice and quick.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-10 15:54:54

So, looks like it's not possible to use image-builder in its current state with Ubuntu 22.04. Looks like a newer version of openssh comes by default that no longer allows for ssh-rsa keys to be used.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-10 15:58:25

Does anyone know what the process would be for updating the SSH key used by Ansible?
Doesn't look like I can override it so a new release would be required I think.

❤️ jayunit100-pub-chnl-plz

cecile

2022-10-11 22:42:34

@mboersma had a PR to add support for Azure

cecile

2022-10-11 22:43:01

GitHub

Add Ubuntu 22.04 LTS for Azure by mboersma · Pull Request #961 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adds Ubuntu 22.04 LTS build support to Azure. Which issue(s) this PR fixes: Refs #960 Additional context: I've tested this and it builds successfully in our DevO...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/961

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-11 08:34:21

☝️ Ok, I think I might have made progress on this. It looks like updating the version of Ansible (and goss, not sure why) has allowed me to get further. The new Ansible version isn't a major version bump so I assume it'll be ok to get a PR up for a new release.
Once I've confirmed the build finishes as expected I will look at getting a PR put up.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-11 10:53:25

PR to bump Ansible and goss versions:

GitHub

Bump dependency versions by AverageMarcus · Pull Request #993 · kubernetes-sigs/image-builder

Bumps goss and Ansible to latest versions to allow for building Ubuntu 22.04 images (with updated SSH server)

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/993

jayunit100-pub-chnl-plz

2022-10-11 12:26:40

Hi @Marcus Noble just wanted to say thanks for that ... noticed the old ansible version recently also but I was too lazy to do anything about it. And yes ubuntu 22 thank you for that also. Cc @kiran keshavamurthy

👍 kiran keshavamurthy

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-11 12:28:27

I'm just testing out ubuntu 22.04 builds for CAPA and CAPG. If all goes well I'll have another PR up with build tasks for those.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-11 16:33:46

PR adding Ubuntu 22.04 support for AWS and GCP - (requires the above PR to be merged)

GitHub

Added Ubuntu 22.04 builds for AWS and GCP by AverageMarcus · Pull Request #994 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/994

mimmus

2022-10-12 14:09:50

@mimmus has left the channel

knfoo

2022-10-12 14:19:03

It is great to see all the work on adding Ubuntu 22.04 support 🎉
We are using these images with metal3 any work ongoing for the Ubuntu 22.04 for the make target build-qemu-ubuntu-2004-efi so it becomes build-qemu-ubuntu-2204-efi ?
I tried looking for it in the issues list but was not able to

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-13 13:41:12

Hey @knfoo I've only been focussing on AWS and GCP for my changes as it's what we need currently but we'll also be needing the qemu targets soon for use with OpenStack.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-13 13:41:31

I'm hoping the changes are basically the same but haven't tried it out yet.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-13 14:40:30

If you happen to have the time and means to test out this PR I think it might be all that's needed:
I'm currently unable to test locally on my ARM-based Mac 😞

GitHub

Added QEMU ubuntu 22.04 build by AverageMarcus · Pull Request #995 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/995

knfoo

2022-10-14 07:42:17

Hey @Marcus Noble
This is great - i will try it out after my vacation.

:thank_you: Marcus Noble

Joe Kratzat

2022-10-18 02:04:08

I was attempting to build a windows image today and I’m getting the following error:

fatal: [default]: FAILED! => {"attempts": 5, "changed": false, "dest": "c:\k\nssm.exe", "elapsed": 0.21601959999999998, "msg": "Error downloading '' to 'c:\k\nssm.exe': The remote server returned an error: (404) Not Found.", "status_code": 404, "url": ""}

When I attempt to go to I see the 404. It seems the URL is set here

What am I doing wrong?

Joe Kratzat

2022-10-18 17:43:11

hmmm seems like that url is bad. I downloaded nssm.exe and placed it in another public place and the image build worked

jsturtevant

2022-10-18 19:19:23

we are aware of this, you have found the mitigation. We are working on addressing it

Joe Kratzat

2022-10-18 19:20:21

awesome thank you. I’ll use the mitigation simple_smile

jsturtevant

2022-10-18 19:46:17

I know you saw it on the other thread, dropping this here for posterity:

GitHub

Update nssm default download location by jsturtevant · Pull Request #999 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/999

:thank_you: Joe Kratzat

Ivan Šumak

2022-10-19 19:28:01

@Ivan Šumak has joined the channel

Portia

2022-10-20 00:05:10

@Portia has joined the channel

Joe Kratzat

2022-10-20 13:46:45

Hey folks.

I’m trying to understand how passwords are set for winRM for CAPI provider images. I see the OVA and VBOX are hardcoding winrm_password in their windows packer config, but AWS and Azure aren’t. How are these later images setting the passwords at image build time?

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-24 07:02:26

Is anyone free to take a look at for me please? 🙏

GitHub

Added Ubuntu 22.04 builds for AWS and GCP by AverageMarcus · Pull Request #994 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/994

Vignesh Goutham

2022-10-24 18:49:32

Hello all, whats the current topic request format for office hours? If there is a doc, can someone point me to it? This one seems to be old. Or do we just post here the day before?

Joe Kratzat

2022-10-24 20:43:00

Hey all, I’m not sure what I’ve done to have the pull-ova-all tests failing in my pr

but I’m seeing the following

 'packer' has been installed to /home/prow/go/src/sigs.k8s.io/image-builder/images/capi/.local/bin, make sure this directory is in your $PATH
hack/ensure-goss.sh
/root/.packer.d/plugins/packer-provisioner-goss: OK
hack/ensure-ovftool.sh
rockylinux-8: FAILED. See logs in the artifacts folder.
ubuntu-2004: FAILED. See logs in the artifacts folder.
photon-3: FAILED. See logs in the artifacts folder.
flatcar: FAILED. See logs in the artifacts folder.
ubuntu-1804: FAILED. See logs in the artifacts folder.
centos-7: FAILED. See logs in the artifacts folder.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:-:-- --:-:-- --:-:--     0
  0     0    0     0    0     0      0      0 --:-:-- --:-:-- --:-:--     0

  1 7400k    1  111k    0     0   204k      0  0:00:36 --:-:--  0:00:36  204k
100 7400k  100 7400k    0     0  7815k      0 --:-:-- --:-:-- --:-:-- 17.7M
govc: Post "": dial tcp 10.2.224.4:443: i/o timeout

TIA for any help simple_smile

Joe Kratzat

2022-10-24 21:43:16

looks like maybe the centos-7 artifact failed in the pull-azure-sigs test https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-bui[…]igs/1584616530543382528/artifacts/azure-sigs/centos-7.log as well

Chris Privitere

2022-10-25 15:55:25

@Chris Privitere has left the channel

Jamie Monserrate

2022-10-27 02:39:37

@Jamie Monserrate has left the channel

Remi Le Trocquer

2022-10-27 09:44:01

@Remi Le Trocquer has joined the channel

Sriraman Srinivasan

2022-10-28 06:45:18

@kiran keshavamurthy Reg: PR 1003, the new ubuntu versions use cloud init to boot and it expects the drive to be labeled as cidata to work. But for some reason setting floppylabel to the value does not seem to take effect. As a result I see only loading the kickstart files (meta-data and user-data) as files on cdrom(able to set the cdlabel tag and gets reflected). However for this to work either xorriso or mkisofs needs to be available to create the cdrom which then can be mounted. Adding either of those will need to change prow job container running the build. So need couple of inputs here

Is it ok to go ahead with using cdrom as mount device for kiskstart files or do you see any other way to do it

Also can you please point me to the prow job where the image will be build

GitHub

Add support for Ubuntu 2204 for vSphere by sriramandev · Pull Request #1003 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1003

image.png

Sriraman Srinivasan

2022-11-04 04:14:42

@kiran keshavamurthy can you please suggest here.

kiran keshavamurthy

2022-11-14 17:42:01

Thanks for the patience @Sriraman Srinivasan. Been a bit busy and was on PTO. Will look into this soon.

👍 Sriraman Srinivasan

k8scapv

2022-10-28 12:42:17

@k8scapv has joined the channel

k8scapv

2022-10-28 13:13:50

any readily available rhel 8 ova vm template for capv ?

swan

2022-10-28 16:13:17

Hey Folks, wanted to know if there is something changed recently from v0.1.13 regarding AMI generation? As the names that are now generated for CAPA AMIs by image builder are as capa-ami-amazon-2-v1.25.3-1665727783 (notice suffix v in k8s release) as opposed to what it was in previous releases capa-ami-amazon-2-1.25.2-00-1664536077 . This is causing issues in CAPA for pushing AMI images, as there is some strict checking of formatting. We could resolve this on CAPA side, but just wanted to make sure if this change was intentional or was done by mistake?

swan

2022-10-28 16:13:50

cc @Skarlso @richcase @Daniel Lipovetsky

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-28 16:15:32

I'm pretty sure it was intentional to be inline with other providers. I recently looked up the change. Give me a min and I'll dig it out

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-28 16:16:55

Released in

GitHub

use kubernetes_semver in AMI naming · kubernetes-sigs/image-builder@05eff6b

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/commit/05eff6b6dcc75e158b51c26d5628ecf4de34152e

GitHub

Release v0.1.13 · kubernetes-sigs/image-builder

v0.1.13

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.13

swan

2022-10-28 16:17:48

thanks @Marcus Noble this helps, I would make change in CAPA accordingly.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-28 16:18:53

Yeah, I already approved your PR in CAPA 🙂

swan

2022-10-28 16:19:06

needs more change 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2022-10-28 16:19:27

Ping me when done 🙂 Happy to review again

swan

2022-10-28 16:19:41

thanks a lot, surely will do 🙂

Peter Bücker

2022-11-08 21:11:36

@Peter Bücker has joined the channel

k8scapv

2022-11-09 05:55:05

can we use existing OVA image (already existing one which is verified by security team) instead of iso and install addons (k8s,cni,container stuff to make it capi v conformant image) on top of that OVA using image builder and create a vm template out of it.Can some one guide me on this

subhasmita

2022-11-09 06:34:19

@subhasmita has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-10 13:51:51

Do the image-builder office hours still happen? Doesn't look like there's been one since July according to the meeting notes.

Joe Kratzat

2022-11-10 13:55:56

I believe if people don’t put things on the agenda they will cancel the meeting

If no Agenda items are present the night before, the meeting will be canceled.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-10 13:57:57

I haven't seen any messages about it being canceled either though. Which makes me wonder if it's just kinda died off.

Joe Kratzat

2022-11-10 13:59:42

yeah I’m not sure there. Last thing about office hours was from

Cecile Robert-Michon (https://kubernetes.slack.com/team/U98JPHB2M)

📣 Hi all, since the last couple of image-builder office hours have had very low attendance and no agenda topics we decided to try something new. We’ll add the agenda entry to the notes a week before the next meeting so folks can add topics throughout the week (just added the one for next week in <a href="https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit#">Image Builder Office Hours - Google Docs</a>) and if there are no topics added by the night before the meeting we’ll cancel the meeting occurrence. Let’s try this out a couple of times and see how it works out.

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1625675974048600

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-10 14:07:39

I might try adding topics for next week and see what happens I guess 😆

😅 Joe Kratzat

cecile

2022-11-10 23:19:10

@codenrhoden used to host these but he had to step away from the project

cecile

2022-11-10 23:20:04

@mboersma @jsturtevant @kiran keshavamurthy and I have been maintaining the project on a best effort basis (basically just keeping the lights on)

cecile

2022-11-10 23:20:27

I personally have a conflict with the current 8am time and can’t make it most weeks

cecile

2022-11-10 23:21:30

the project is a bit low on maintainers right now so if anyone has interest in stepping up to host the office hours please do

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-11 07:14:07

That's actually one of the things I wanted to discuss 🙂 image-builder is a pretty important project for us at Giant Swarm and would love if we could help out . There's a lot of improvements / fixes we'd like to see but the current level of activity on the project makes that difficult. I'll get some topics added to the agenda for next week later today 🙂

❤️ jsturtevant

richcase

2022-11-14 07:29:26

We are in a similar position as well, image builder is pretty important to us as well. We also want to make improvements/changes.

🎉 Marcus Noble

kiran keshavamurthy

2022-11-14 17:39:35

Image-builder is important to us as well. I have to split time between upstream and downstream work so sometimes upstream work takes a hit.
I’m completely on board with changing the office hours to better suit everyone and love to get more maintainers

👍 cecile, jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 16:01:17

Just wanted to double check others are joining the office hours. I'm currently waiting in the call. 🙂

:this: MarcelMue

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 16:07:37

@kiran keshavamurthy @richcase @cecile Are any of you planning to join?

richcase

2022-11-17 16:08:07

I am, sorry

cecile

2022-11-18 00:07:20

I personally have not had the bandwidth to be active enough as image-builder maintainer lately and I think it's time I officially step down to make space for new folks

cecile

2022-11-18 00:13:21

I will make a PR soon to officially move myself to emeritus. I also nominate @mboersma as new maintainer, he's been a lot more involved in the project than I have lately and is interested in helping. cc @kiran keshavamurthy

:ack: kiran keshavamurthy

jsturtevant

2022-11-21 16:58:55

I can no longer make the 8am meetings on Thursdays but would be open to meeting another time. Should we get a doodle poll going for new time as it seems we have a bunch on new folks?

cecile

2022-12-01 22:32:36

Marcus Noble (k8s@marcusnoble.co.uk)

2022-12-01 22:54:05

Thanks @cecile 🙂 I've opened a PR to add myself to the reviewers:

GitHub

Add @AverageMarcus as reviewer by AverageMarcus · Pull Request #1025 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1025

🎉 cecile

aniruddha

2022-11-12 05:13:18

@aniruddha has left the channel

Kemalettin

2022-11-12 15:14:33

@Kemalettin has joined the channel

Magnus RC

2022-11-14 12:39:24

@Magnus RC has joined the channel

willie

2022-11-14 21:33:28

@willie has joined the channel

swan

2022-11-16 11:46:00

Hey folks, since the newer k8s release versions are compatible with containerd version 1.6.4+, image builder still uses default as 1.6.2.
I tried below giving parameters such that we use compatible version of containerd while generating CAPA AMIs:

"containerd_version": "1.6.5",
"containerd_checksum": "cf02a2da998bfcf61727c65ede6f53e89052a68190563a1799a7298b0cea86b4",
"containerd_url": ""

but its failing with below error:

fatal: [default]: FAILED! => {"changed": true, "checksum_dest": null, "checksum_src": "8b354c7fcc59c66ce8ade0bc137782838709fa3c", "dest": "/tmp/containerd.tar.gz", "elapsed": 0, "msg": "The checksum for /tmp/containerd.tar.gz did not match 91f1087d556ecfb1f148743c8ee78213cd19e07c22787dae07fe6b9314bec121; it was cf02a2da998bfcf61727c65ede6f53e89052a68190563a1799a7298b0cea86b4.", "src": "/tmp/.ansible/ansible-tmp-1668598896.3124301-85962-228207629295306/tmpps0mndg2", "url": ""}

Although I see checksum set is correct but i think somehow this is interfering with other attribute related to CRI.

swan

2022-11-16 11:47:32

oh i think the attribute should be containerd_sha256 facepalm

swan

2022-11-16 11:49:18

although at some point, we must update this version to higher containerd version compatible with recent k8s versions

swan

2022-11-16 12:11:18

also even after setting right paramter I am getting below error

  ubuntu-20.04: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to create temporary directory. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"echo /tmp/.ansible\"&& mkdir \"echo /tmp/.ansible/ansible-tmp-1668600507.193056-96724-263289874289379\" && echo ansible-tmp-1668600507.193056-96724-263289874289379=\"echo /tmp/.ansible/ansible-tmp-1668600507.193056-96724-263289874289379\" ), exited with result 1", "unreachable": true}

could someone help solve this

swan

2022-11-17 05:28:46

cc @Marcus Noble if you know about this

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 07:04:15

Sorry, I was AFK yesterday. 🙂 Do you get the same error without making the changes to containerd? I doesn't look like it should be related from what I can tell. I can try and dig in a bit more later today and see if I can see the cause.

richcase

2022-11-17 07:24:59

I can try this morning

swan

2022-11-17 07:25:47

I wanted to change containerd version thats why trying this, but didnt get this error while not specifying containerd versions etc.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 07:26:24

Was the 3 containerd values the only thing you changed?

swan

2022-11-17 07:27:32

{
 "kubernetes_series": "1.25",
 "kubernetes_semver": "1.25.4",
 "kubernetes_rpm_version": "1.25.4-0",
 "kubernetes_deb_version": "1.25.4-00",
 "kubernetes_source_type": "pkg",
 "kubernetes_http_source": "",
 "kubernetes_rpm_repo": "",
 "kubernetes_rpm_gpg_key": "\" \"",
 "kubernetes_rpm_gpg_check": "True",
 "kubernetes_deb_repo": "\" kubernetes-xenial\"",
 "kubernetes_deb_gpg_key": "",
 "kubernetes_container_registry": "registry.k8s.io",
 "kubernetes_load_additional_imgs": "false",
 "kubeadm_template": "etc/kubeadm.yml",
 "containerd_version": "1.6.6",
 "containerd_sha256": "0212869675742081d70600a1afc6cea4388435cc52bf5dc21f4efdcb9a92d2ef",
 "containerd_url": ""
}

This is the env vars i generally use to generate AMIs, only thing added here is containerd stuff

swan

2022-11-17 08:30:06

ok made some progress, looks like i was using wrong containerd url, so removed url and it started working but failed at this point

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'
    flatcar-stable: fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'"}

I am not sure if this version is supported or not, ideally i should be able to build with any containerd version

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 08:39:47

Any idea what task is throwing that error? Struggling to find where the version is checked 😕

swan

2022-11-17 08:40:07

sorry missed it 😄

 TASK [containerd : Copy in containerd config file etc/containerd/config.toml] **

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 08:42:08

😕 That doesn't seem to do anything with the version.

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 08:42:51

Mind posting the ~5 tasks that completed in the run up to that error?

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 08:44:32

I think it might be complaining about a different "version" in that error. As far as I can see, the containerd_version is used for 2 things - building the URL (which you're overwriting anyway) and by goos to check the right version was installed.

swan

2022-11-17 09:01:18

here you go

flatcar-stable: TASK [include_role : containerd] *
    flatcar-stable:
    flatcar-stable: TASK [containerd : download containerd] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Create a directory if it does not exist] *
    flatcar-stable: ok: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : unpack containerd for Flatcar to /opt/bin] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : delete /opt/cni directory] 
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : delete /etc/cni directory] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Creates unit file directory] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Create systemd unit drop-in file for containerd to run from /opt/bin] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Create containerd memory pressure drop in file] 
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Create containerd max tasks drop in file] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Create containerd http proxy conf file if needed] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Creates containerd config directory] *
    flatcar-stable: changed: [default]
    flatcar-stable:
    flatcar-stable: TASK [containerd : Copy in containerd config file etc/containerd/config.toml] 
    flatcar-stable: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'
    flatcar-stable: fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'"}
    flatcar-stable:
    flatcar-stable: PLAY RECAP **
    flatcar-stable: default                    : ok=34   changed=25   unreachable=0    failed=1    skipped=145  rescued=0    ignored=0
    flatcar-stable:

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 09:40:49

ah ha! I think it's this line: https://github.com/kubernetes-sigs/image-builder/blob/02df45969409c7f18f2cf7e63b70[…]i/ansible/roles/containerd/templates/etc/containerd/config.toml
Can you try setting kubernetes_semver to be v1.25.4 (hopefully that doesn't break elsewhere)

GitHub

image-builder/config.toml at 02df45969409c7f18f2cf7e63b70c4dc0e2fa442 · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/02df45969409c7f18f2cf7e63b70c4dc0e2fa442/images/capi/ansible/roles/containerd/templates/etc/containerd/config.toml#L14

swan

2022-11-17 09:41:25

Oh great i will give it a try

swan

2022-11-17 09:54:58

oh atleast the previous failure is gone now 😅 thanks @Marcus Noble I will ping back if I get more errors 😉

Marcus Noble (k8s@marcusnoble.co.uk)

2022-11-17 10:01:57

🎉

k8scapv

2022-11-18 06:50:21

can i run ansible scripts alone on the OVA from the image builder repo ? can some one guide on this.

k8scapv

2022-11-18 06:51:12

I do not want image builder to create an ova for me ,i have an ova and i just want to install rest all things on top of that ova,so that i can use that ova to create vmtemplate

k8scapv

2022-11-18 06:51:25

can some one guide how can i do that

jsturtevant

2022-11-21 17:08:09

there isn't a guide but if you have the vm booted you can configure Ansible to connect to that VM then run the ansible scripts in . You will have to set all the ansible variables yourself

GitHub

image-builder/images/capi/ansible at master · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/tree/master/images/capi/ansible

Julien Klaer

2022-11-22 17:05:37

@Julien Klaer has joined the channel

Joe Kratzat

2022-11-29 13:01:19

Hey all,

I have a PR failing tests around pull-azure-sigs

I see

 sig-flatcar: FAILED. See logs in the artifacts folder.
sig-flatcar-gen2: FAILED. See logs in the artifacts folder.

in the logs, but not seeing why flatcar is failing.

Jeremi Piotrowski

2022-11-29 15:53:43

im on it

:thank_you: Joe Kratzat, mboersma

Joe Kratzat

2022-11-29 19:55:52

any luck?

Jeremi Piotrowski

2022-11-30 11:16:26

it's merged

GitHub

Set FLATCAR_VERSION to latest available in Azure CI by jepio · Pull Request #1021 · kubernetes-sigs/image-builder

What this PR does / why we need it: The default FLATCAR_VERSION=current resolves to something that is not always available on Azure. Override its value to the latest version actually available in A...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1021

Ricky Sadowski

2022-11-30 18:19:37

@Ricky Sadowski has joined the channel

Vibhor Chinda

2022-11-30 18:38:44

@Vibhor Chinda has joined the channel

cpanato

2022-12-02 10:23:11

seeking a lgtm here 🙂

GitHub

update capg images to use latest k8s releases by cpanato · Pull Request #1019 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1019

👍 jsturtevant

Vibhor Chinda

2022-12-05 16:09:45

Hii everyone :)) 👋
I attended the last capi office hours and found out that this project needs some contributors who can help.

I am a beginner in this space but am looking to contribute wherever and whatever I can.

I would love to know more about the project and can help with beginner friendly issues if there are any 👀.

Any kind of guidance will be great :))
thanks

jsturtevant

2022-12-05 17:15:23

welcome! We have a book to get started: . Triaging issues could be a good way to get started getting familiar with the type of work in the project. Feel free to ask questions

:ty: Vibhor Chinda

Vibhor Chinda

2022-12-06 06:37:14

Thanks @jsturtevant
I will have a look at the book :))

Vivek Koppuru

2022-12-07 08:33:19

@Vivek Koppuru has joined the channel

Esme

2022-12-16 01:16:16

@Esme has joined the channel

swan

2022-12-19 13:45:35

Hey folks, I am trying to build CAPA AMI images, but I am getting following ansible ssh error, could someone help around fixing this issue? This has been specifically started occurring recently as it used to work before. Maybe it has something to do with mac OS update, but I am not sure

amazon-2: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Unable to negotiate with 127.0.0.1 port 55506: no matching host key type found. Their offer: ssh-rsa", "unreachable": true}

nikparasyr

2022-12-19 13:46:25

there are a couple of issues and PRs open for this

swan

2022-12-19 13:47:04

oh thats good to know, will wait for the fix. Thankyou @nikparasyr

nikparasyr

2022-12-19 13:47:23

maybe this is the one blocking you?

GitHub

Workaround SSH error "no matching host key type found. Their offer: ssh-rsa" · Issue #905 · kubernetes-sigs/image-builder

What steps did you take and what happened: Right now, if one builds the CAPI images with OpenSSH version 8.8+ on the host system, the environment variable below is required to make provisioning wor...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/905

swan

2022-12-19 13:47:44

yes that’s right

swan

2022-12-19 13:48:18

I ll give it a try with args

nikparasyr

2022-12-19 13:48:26

there is a comment there on how to bypass it. The PRs to fix it have been open for a while AFAIK, so i just had to use what is mentioned in the comment to move forward. Hope it works for you

👍 swan

Sai Preetham Bojja

2022-12-19 15:56:20

@Sai Preetham Bojja has joined the channel

Alex Hernandez

2022-12-19 15:57:47

@Alex Hernandez has joined the channel

quba

2022-12-28 16:10:28

@quba has joined the channel

bkc

2022-12-30 15:59:51

@bkc has joined the channel

Aravind Ravichandran

2023-01-04 10:27:09

@Aravind Ravichandran has joined the channel

Aravind Ravichandran

2023-01-04 10:27:58

Hi team , need help on creating clusterapi image for kubevirt

Julien Klaer

2023-01-05 08:37:05

Hello there 👋 Happy new year!
Is there any chance this issue makes sense? Would it make sense to define the end goal of it? I can try to work something out for this

jsturtevant

2023-01-09 17:03:01

this would avoid passing it to the image builder, but would still need it in scripts? I think it probably makes sense to minimize passing hte variables around. @mboersma any thoughts?

jdetiber

2023-01-12 15:42:27

@jdetiber has left the channel

Paokrab

2023-01-16 17:35:05

@Paokrab has joined the channel

Amim Knabben

2023-01-17 14:58:20

Hey, did already tried to run image-builder on a pre-existent image? More specifically decouple and reuse the ansible roles on a running VM.

vrabbi

2023-01-17 17:21:35

Ive done it

:partyk8s: Amim Knabben

Alessandro Giorgio Togna

2023-01-19 21:22:26

@Alessandro Giorgio Togna has joined the channel

Thorben

2023-01-21 13:06:53

@Thorben has joined the channel

Thorben

2023-01-21 13:14:07

Hey! With the latest minor releases of the 24, 25 and 26 series I'm running into build errors for GCP cluster-api images. 1.24.9, 1.25.5 and 1.26.0 work just fine, with 1.24.10, 1.25.6 and 1.26.1 I get the same error across all 3 builds:

ubuntu-2204: TASK [kubernetes : Install Kubernetes] **
    ubuntu-2204: fatal: [default]: FAILED! => {"cache_update_time": 1674304180, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"       install 'kubelet=1.24.10-00' 'kubeadm=1.24.10-00' 'kubectl=1.24.10-00' 'kubernetes-cni=1.1.1-00'' failed: E: Unable to correct problems, you have held broken packages.\n", "rc": 100, "stderr": "E: Unable to correct problems, you have held broken packages.\n", "stderr_lines": ["E: Unable to correct problems, you have held broken packages."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSome packages could not be installed. This may mean that you have\nrequested an impossible situation or if you are using the unstable\ndistribution that some required packages have not yet been created\nor been moved out of Incoming.\nThe following information may help to resolve the situation:\n\nThe following packages have unmet dependencies:\n kubeadm : Depends: kubernetes-cni (>= 1.2.0)\n kubelet : Depends: kubernetes-cni (>= 1.2.0)\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Some packages could not be installed. This may mean that you have", "requested an impossible situation or if you are using the unstable", "distribution that some required packages have not yet been created", "or been moved out of Incoming.", "The following information may help to resolve the situation:", "", "The following packages have unmet dependencies:", " kubeadm : Depends: kubernetes-cni (>= 1.2.0)", " kubelet : Depends: kubernetes-cni (>= 1.2.0)"]}

I'm assuming other providers are also affected, but couldn't find anything in the issues. That is, if this file is actually responsible for installing the kubernetes-cni .deb package:

{
  ...
  "kubernetes_cni_deb_version": "1.1.1-00",
  ---
}

Is there something I'm doing wrong or did this simply not pop up in the nightly CI yet?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-01-23 07:28:21

I was able to build it with these values:

{
"kubernetes_cni_deb_version": "**", 
"build_name": "ubuntu-2204",
"distribution_release": "jammy",
"distribution_version": "2204"
}

This ensures the latest deb version is used from the package manager.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-01-23 07:32:55

The latest nightly run is here: Prow Job
This runs this script: ci-gce-nightly.sh
I don't know much about it but it seems to use this for the 1.26 builds which is still targeting the 1.26.0 version - overwrite-1-26.json

Thorben

2023-02-12 10:00:05

Issue has been resolved in this commit, by the looks of it - I can build the versions mentioned initially now:

puneetk

2023-01-26 06:28:22

@puneetk has joined the channel

Sakari Poussa

2023-01-26 13:57:26

@Sakari Poussa has joined the channel

Joe Kratzat

2023-02-08 13:55:14

We have run into a few times where users are getting errors because the IBVERSION ENV isn’t set when trying to build an image for our packer config. I see other configs using the same “ib_version”: “{{envIBVERSION}}“, but nothing in the book calls out setting this. How are others 1) setting this environment variable for the user or 2) informing the users to set it?

Jon Zeolla

2023-02-14 18:05:55

@Jon Zeolla has joined the channel

tomoyuki

2023-02-15 08:00:17

@tomoyuki has joined the channel

2023-02-23 08:11:24

@fc has joined the channel

2023-02-23 08:45:16

Hi Everyone,

I just started experimenti with image-builder to build our own CAPZ Flatcar images.

for capz the init-sig.sh script is run to create the image definition but the format is very hardcoded and not ideal ( at least for our use case )

the generated format is capi-flatcar-stable-$(FLATCARVERSION)-gen2 while ideally i would like, similar to what the CAPI images are capi-flatcar-stable-$(KUBERNETESVERSION) so i can then have an image inside of it for each version of flatcar

➜ az sig image-definition list-community --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --location westeurope | jq '.[].name'
"flatcar-stable-amd64-capi-v1.23.13"
"flatcar-stable-amd64-capi-v1.24.6"
"flatcar-stable-amd64-capi-v1.24.9"
"flatcar-stable-amd64-capi-v1.25.4" 
"flatcar-stable-amd64-capi-v1.25.6"
"flatcar-stable-amd64-capi-v1.26.0"
 
➜ az sig image-version list-community --location westeurope --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --only-show-errors --gallery-image-definition flatcar-stable-amd64-capi-v1.24.9 -o table
ExcludeFromLatest    Location    Name      PublishedDate                     UniqueId
-------------------  ----------  --------  --------------------------------  --------------------------------------------------------------------------------------------------------------------------------
True                 westeurope  3374.2.1  2023-01-06T00:29:51.344093+00:00  /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.24.9/Versions/3374.2.1

how does the CAPI project achieve this using the same init-sig.sh script that is in the repo ?

thanks !

2023-02-23 09:25:47

@cecile in case you know about it , i did not want to cross post in cluster-api-azure

🙏

2023-02-23 11:43:33

for reference

Jeremi Piotrowski

2023-02-24 14:58:55

hi @fc. what we do for Flatcar is we use image-builder to build the image, and then republish into a seperate gallery that has the desired structure that we want. The republishing is done by using the gallery-image-version-id as the image source

2023-02-27 08:10:54

I see, thanks !

2023-02-27 15:45:28

@Jeremi Piotrowski i have a follow up question to this 🙂

I am building directly with image-builder into a Community gallery and is working fine now but , in order to use my image, i need to specify something like

      image:
        computeGallery:
          gallery: test-xxxx-820f-b52ca78f96e6
          name: capi-flatcar-stable-1.24.9-gen2
          plan:
            offer: flatcar-container-linux-free
            publisher: kinvolk
            sku: stable-gen2
          version: latest

but when i use the flatcar4capi gallery i don't have to specify the plan section.

is this difference because of your process of republishing into a separate gallery ?

thanks

Jeremi Piotrowski

2023-02-27 15:52:37

@Mateusz Gozdek (invidian) there is some magic here right?

2023-02-27 15:53:07

i like magic 🎉

2023-02-27 15:53:10

🙏

Jeremi Piotrowski

2023-02-27 15:53:16

or do we build the community gallery image with a community gallery image as a source?

2023-02-27 15:54:09

my concern is that

i did a lot of testing using the flatcar4capi images

i am moving onto building my images from image-builder

to use flatcar4capi images i don't need to accept terms of the parent image like i do with my build ( and i have to accept the parent flatcar terms ... which is ok , but is different and i want to understand why )

maybe they are actually not equivalent somehow ?

Mateusz Gozdek (invidian)

2023-02-27 16:22:05

Hello. Images in flatcar4capi are build from Flatcar VHDs imported into a SIG, so their advantage is that they don't require plan information. That's the big part of it.

There is also flatcar community gallery, which you can use as a source for building your images using image-builder. Let me dig up some sample JSON packer values we use for building the images.

2023-02-27 16:22:47

yeah that make sense. thanks

There is also flatcar community gallery, which you can use as a source for building your images using image-builder. Let me dig up some sample JSON packer values we use for building the images.

that is what i am doing 👍 through image-builder

Mateusz Gozdek (invidian)

2023-02-27 16:23:08

From our release automation:

  cat <{
  "sig_image_version": "${FLATCAR_VERSION}",
  "kubernetes_semver": "${KUBERNETES_SEMVER}",
  "image_name": "${IMAGE_NAME}",
  "image_offer": "",
  "image_publisher": "",
  "image_sku": "",
  "image_version": "",
  "plan_image_offer": "",
  "plan_image_publisher": "",
  "plan_image_sku": "",
  "source_sig_subscription_id": "${AZURE_SUBSCRIPTION_ID}",
  "source_sig_resource_group_name": "${STAGING_SIG_RESOURCE_GROUP}",
  "source_sig_name": "${FLATCAR_STAGING_GALLERY_NAME}",
  "source_sig_image_name": "${FLATCAR_IMAGE_NAME}",
  "source_sig_image_version": "${FLATCAR_VERSION}"
}
EOF

2023-02-27 16:24:18

now i am just trying to validate the differences between the nodes i get with flatcar4capi and from image-builder
built images.

I can see that with the image from flatcar4capi i get containerd 1.6.14 while on the one from image-builder i get 1.6.2 ... do you customize / override that ?

Mateusz Gozdek (invidian)

2023-02-27 16:25:35

No, we build from master branch of image-builder. I don't remember, but maybe in Flatcar it's using baked in containerd version as opposed to one installed by Ansible? So maybe the difference is Flatcar version used?

2023-02-27 16:27:11

thanks for confirming that.

i am going through the ansible code and our pipelines now , maybe we are just out of date or something.

i can see the playbook eventually run does indeed specify 1.6.2

 sig-flatcar-gen2: Executing Ansible: ansible-playbook -e packer_build_name="sig-flatcar-gen2" -e packer_builder_type=azure-arm --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=91f1087d556ecfb1f148743c8ee78213cd19e07c22787dae07fe6b9314bec121 pause_image=k8s.gcr.io/pause:3.6 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.6.2

now i just need to walk back and find where and why those are set to 1.6.2 🙂

thanks foryour help 🙏

👍 Mateusz Gozdek (invidian)

2023-02-27 16:34:58

simple, i am building from tag 01.13 ( which indeed uses containerd 1.6.2 still ) and master is using the upgraded version

🙂 thanks again

👌 Mateusz Gozdek (invidian)

Mateusz Gozdek (invidian)

2023-02-27 16:37:51

Cool. Feel free to ping me if you have some further questions. I was mainly driving the work on community SIG for Flatcar and I'm always happy to help (or redirect to team members which might be more knowledgeable than me) 🙂

2023-02-27 16:38:18

thanks will do 🙏

2023-02-28 11:37:54

hi @Mateusz Gozdek (invidian) sorry to ping you again, one quesiton

you said you build VHD using image-builder then import into SIG

i wanted to try the same but when i look at i don't see flatcar in the lsit of VHD supported targets.

do you have patch or something on top of image builder ?

if your code for the build pipeliens is available on github i'ld love to take a look 🙂
thanks 🙏

Mateusz Gozdek (invidian)

2023-02-28 11:45:43

The script is not public yet, as it's a first version and we didn't put much effort into it yet, but yeah, eventually it will be public. This is how it looks: .

As you won't have access to storage account which holds Flatcar VHD images, You probably need to download a VHD from the Flatcar release, upload it to storage account, and then you should be able to use the script above.

It's nothing fancy really.

2023-02-28 11:46:04

thanks 🙏

2023-02-28 17:42:50

unrelated , to an extent , question.

do you know why the .4 of the stable channel does not exist in azure as a vm image yet ?

➜ az vm image list --publisher kinvolk --sku stable-gen2 --offer flatcar-container-linux-free --all -o json | jq -r '[.[].version] | sort_by( values | split(".") | map(tonumber) ) | .[-1]'
3374.2.3

Mateusz Gozdek (invidian)

2023-02-28 17:46:36

I guess the image has not been approved yet, which is odd. Maybe @Kai Lüke or @Jeremi Piotrowski know what's the status of it?

BTW, is the image available in community gallery? I can make it so if it's also missing.

2023-02-28 17:47:36

have not checked but can do that

2023-02-28 17:49:04

in the flatcar4capi i can only see .1 , let me see to find the name of the flatcar community gallery

2023-02-28 17:52:41

in the flatcar one i see the same, .3 as latest

 λ az sig image-version list-community --location westeurope --public-gallery-name flatcar-23485951-527a-48d6-9d11-6931ff0afc2e --only-show-errors --gallery-image-definition flatcar-stable-amd64 | jq '.[].name'
"3374.2.0"
"3374.2.1"
"3374.2.3"

👍 Mateusz Gozdek (invidian)

Mateusz Gozdek (invidian)

2023-02-28 17:53:55

I'll try to push the latest image there.

2023-02-28 17:54:34

nice thanks 🙏 i will need to see to change my pipeline to use the gallery rather than the vm image as source

👍 Mateusz Gozdek (invidian)

Mateusz Gozdek (invidian)

2023-02-28 18:34:44

.4 image should be there already.

2023-02-28 18:35:06

Thanks 🙏

Jeremi Piotrowski

2023-02-28 18:41:43

Yeah azure publishing got delayed this time round and is taking longer as well

2023-03-01 10:38:17

@Mateusz Gozdek (invidian) thanks for all your help, i switched to the community gallery "/CommunityGalleries/flatcar-23485951-527a-48d6-9d11-6931ff0afc2e/Images/flatcar-stable-amd64/Versions/3374.2.4"

just a question, how official is this gallery when compared to the flatcar Marketplace offer ?

Mateusz Gozdek (invidian)

2023-03-01 10:42:49

It's official (), but you would probably be an early adopter. We hope it will enable Flatcar users on Azure to get faster access to latest image versions (because of easier and more automated release process) and without requirement of accepting plans.

Right now image publishing there is not wired to the CI, so there might be still a delay until the images show up, but when we see people using it, I'm sure CI will be prioritized to have the process fully automated.

GitHub

[RFE] Offer Flatcar (and CAPI Flatcar) images via Azure Community Gallery · Issue #734 · flatcar/Flatcar

Current situation Azure has recently put into public preview Community Galleries support for Compute galleries, which allows sharing the images between subscriptions and tenants and use existing SI...

Original URL: https://github.com/flatcar/Flatcar/issues/734

2023-03-01 10:45:44

nice thanks

without requirement of accepting plans.

so using the Community Gallery as my source rather than the Martketplace offer means i don't have to accept terms in every subscription ? nice

We hope it will enable Flatcar users on Azure to get faster access to latest image versions

👍 exactly why i switched

I am ok to be an early adopter as we are not in production with capz yet and i'd rather switch the sooner the better and give feedback if needed

Right now image publishing there is not wired to the CI
'm sure CI will be prioritized to have the process fully automated.

🙏 🤞

I might come here and ask for new images when i see releases in the flatcar RSS feed 🙂

thanks

Mateusz Gozdek (invidian)

2023-03-01 10:47:38

I'll make sure images are published as soon as I see a feed as well then. It's probably time to enable replication to all regions then as well.

👍 fc

2023-03-01 10:48:03

Right now image publishing there is not wired to the CI

I do wonder though if ,when that time comes, i should stop building my own images ... i mean here https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/2890/files?short_path=91b8a4f#diff-91b8a4f39cd0f7ee28f[…]67069103bd0fa528e84cb4d3 you do mention that those are just reference images ... but i am literally just rebuilding the same thing 🙂

Mateusz Gozdek (invidian)

2023-03-01 10:50:55

Yes, CAPI images we publish are reference images for testing and CI use with no regular/security updates guarantees. This is the same for all CAPI images available from the maintainers. So users are recommended to build their own images with versions they need etc.

🙏 fc

2023-03-09 08:20:59

@Mateusz Gozdek (invidian) i feel really bad for asking for this ... but ... any chance you could push 3374.2.5 to flatcar community gallery ? 🙏

Mateusz Gozdek (invidian)

2023-03-09 10:12:09

Sure, it's crunching. I had a day off yesterday 🙂

2023-03-09 10:13:10

🙏 Thanks , very appreciated ... is the last bit i need to check my pipeline will pick it up tonight 🤞

Саша Рудан (Sasha Rudan)

2023-02-24 01:24:11

@Саша Рудан (Sasha Rudan) has joined the channel

Danny Bessems

2023-02-26 09:49:41

@Danny Bessems has joined the channel

Slackbot

2023-02-26 09:49:41

This message was deleted.

Shyam P R

2023-02-27 01:26:49

it may be helpful for folks to know the provider which you are trying

Jhonathan Cavalcante

2023-02-28 02:48:33

@Jhonathan Cavalcante has joined the channel

Kai Lüke

2023-03-01 03:20:08

@Kai Lüke has joined the channel

2023-03-01 09:01:04

Hi Everyone !

I have a question about releases on

the last release is 0.1.12 from May 2022, there was a tag 0.1.13 from Sep 2022 ( which is what i am using for my builds )

but since then a lot of changes have been pushed, including some that are a requirement for recent versions of kubernetes like ( )

Do you think is worth cutting a new release to also limit the amount of changes in between releases ?

Danny Bessems

2023-03-02 08:28:06

Similarly, the latest container image for the image-builder is ancient as well, it is missing all the latest make targets (for instance no Ubuntu 22.04).

2023-03-02 08:30:08

and, i did not check but i hope , that AZ cli will be more up-to-date in the master branch ( or i will do a PR for it ) since a bunch of commands fail with current version 🙂

Danny Bessems

2023-03-02 08:32:45

Do you know what version is in the tarball that is linked in the CAPI Image-builder documentation? Is that then 0.1.12 or does it include all commits since?

2023-03-02 08:33:35

no i don't know, but i don't use 0.1.12 i use the tag 0.1.13 ( which did not get a release though not sure why )

i know flatcar upstream uses master ...

Danny Bessems

2023-03-02 08:34:31

Oh my, that could explain my struggles getting a working Ubuntu OVA. I'll have to check...

🤞 fc

Danny Bessems

2023-03-02 08:38:20

Yeah, the tarball is based on master including all commits (it uses github's tarball api endpoint)

Yike Wang

2023-03-02 11:54:10

so.. will we have a new release recently?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-02 13:28:36

@kiran keshavamurthy @mboersma Do you think we could get a new release cut sometime soon?

kiran keshavamurthy

2023-03-02 18:28:54

Yes, I can create a new tag today or tomorrow. Do we need any open PRs to get in before creating a new tag?

🙏 fc

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-02 19:55:25

@fc I think you still have one open that would be good to go in don’t you?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-02 19:56:19

Just need an approval https://github.com/kubernetes-sigs/image-builder/pull/1087

GitHub

Do not use same variable for managed_image_storage_account_type and SIG storage_account_type by primeroz · Pull Request #1087 · kubernetes-sigs/image-builder

What this PR does / why we need it: SIG storage_account_type support Standard_ZFS but managed_image_storage_account_type does not, using the same variable means we can't use Zone redundancy for SIG...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1087

2023-03-02 19:56:33

Yes , it got assigned and is a very simple one so hopefully it will make it 👍

jsturtevant

2023-03-03 00:00:29

this one too:

GitHub

Updating cloudbase-init version to v1.1.4 by marosset · Pull Request #1091 · kubernetes-sigs/image-builder

What this PR does / why we need it: Cloudbase-init v1.1.4 contains a fix for cloudbase/cloudbase-init#94 which impacts node provisioning on Azure / with CAPZ Additional context Add any other contex...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1091

jsturtevant

2023-03-03 00:00:57

@kiran keshavamurthy Is there docs or maybe we can pair for the release? I don't know that process

kiran keshavamurthy

2023-03-07 01:04:19

@jsturtevant As discussed I’ll update the docs on how to tag and release

:ty: jsturtevant

kiran keshavamurthy

2023-03-07 01:05:33

I’ve created a tag for v0.1.14 today.
@Sanika Gawhane Can you create a PR to promote the container image from staging to prod please?

:ack: Sanika Gawhane, jsturtevant

:ty: jsturtevant

2023-03-07 15:29:05

thanks 👍

Yike Wang

2023-03-02 11:51:17

@Yike Wang has joined the channel

Yike Wang

2023-03-06 15:48:24

Hi everyone, did you ever see such error when make build-ami-flatcar flatcar ami? Any guidance for the checkings? I am using v0.1.13 branch. cc @swan

    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : unpack crictl] *
    amazon-ebs.{{user build&#95;name}}: changed: [default]
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Remove crictl tarball] *
    amazon-ebs.{{user build&#95;name}}: changed: [default]
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Create kubelet default config file] *
    amazon-ebs.{{user build&#95;name}}: changed: [default]
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Enable kubelet service] *
    amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service kubelet: host"}
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: PLAY RECAP **
    amazon-ebs.{{user build&#95;name}}: default                    : ok=45   changed=34   unreachable=0    failed=1    skipped=166  rescued=0    ignored=0
    amazon-ebs.{{user build&#95;name}}:

Jeremi Piotrowski

2023-03-06 17:13:06

taking a look

Jeremi Piotrowski

2023-03-06 17:51:01

having trouble running AMI build right now, do you have a full log?

Jeremi Piotrowski

2023-03-06 17:51:20

and have you tried HEAD of the master branch?

Yike Wang

2023-03-07 06:46:52

full log on v0.1.12 (should be same with 0.1.13):

Yike Wang

2023-03-07 06:47:13

*Thread Reply:* None

Untitled

Yike Wang

2023-03-07 06:49:28

I tried on master or v0.1.14, it is even worse with "ansible-playbook: error: argument --scp-extra-args: expected one argument":

no_proxy=** make build-ami-flatcar
hack/ensure-ansible.sh
Starting galaxy collection install process
....
==> amazon-ebs.{{user build&#95;name}}: Executing Ansible: ansible-playbook -e packer_build_name="flatcar-stable" -e packer_builder_type=amazon-ebs --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=8e227caa318faa136e4387ffd6f96baeaad5582d176202fe9da69cde87036033 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.6.8 containerd_wasm_shims_url= containerd_wasm_shims_version=v0.3.3 containerd_wasm_shims_sha256=da84b1c065a58f95a841d39e143cd7115d43e6faedcce7a8782f2942388260d7 containerd_wasm_shims_runtimes="" crictl_url= crictl_sha256= crictl_source_type=http custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_http_source= kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key=" " kubernetes_rpm_gpg_check=True kubernetes_deb_repo=" kubernetes-xenial" kubernetes_deb_gpg_key= kubernetes_cni_deb_version=1.2.0-00 kubernetes_cni_rpm_version=1.2.0-0 kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=http kubernetes_semver=v1.26.2 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.26.2-00 kubernetes_rpm_version=1.26.2-0 no_proxy= pip_conf_file= python_path=/opt/bin/builder-env/site-packages redhat_epel_rpm= epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/etc/systemd sysusr_prefix=/opt sysusrlocal_prefix=/opt load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= build_target=virt amazon_ssm_agent_rpm= --extra-vars ansible_python_interpreter=/opt/bin/python --extra-vars  --scp-extra-args -O -e ansible_ssh_private_key_file=/var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/ansible-key3776119403 -i /var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/packer-provisioner-ansible3049382186 /Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/ansible/node.yml
    amazon-ebs.{{user build&#95;name}}: usage: ansible-playbook [-h] [--version] [-v] [-k]
    amazon-ebs.{{user build&#95;name}}:                         [--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER]
    amazon-ebs.{{user build&#95;name}}:                         [-c CONNECTION] [-T TIMEOUT]
    amazon-ebs.{{user build&#95;name}}:                         [--ssh-common-args SSH_COMMON_ARGS]
    amazon-ebs.{{user build&#95;name}}:                         [--sftp-extra-args SFTP_EXTRA_ARGS]
    amazon-ebs.{{user build&#95;name}}:                         [--scp-extra-args SCP_EXTRA_ARGS]
    amazon-ebs.{{user build&#95;name}}:                         [--ssh-extra-args SSH_EXTRA_ARGS] [--force-handlers]
    amazon-ebs.{{user build&#95;name}}:                         [--flush-cache] [-b] [--become-method BECOME_METHOD]
    amazon-ebs.{{user build&#95;name}}:                         [--become-user BECOME_USER] [-K] [-t TAGS]
    amazon-ebs.{{user build&#95;name}}:                         [--skip-tags SKIP_TAGS] [-C] [--syntax-check] [-D]
    amazon-ebs.{{user build&#95;name}}:                         [-i INVENTORY] [--list-hosts] [-l SUBSET]
    amazon-ebs.{{user build&#95;name}}:                         [-e EXTRA_VARS] [--vault-id VAULT_IDS]
    amazon-ebs.{{user build&#95;name}}:                         [--ask-vault-password | --vault-password-file VAULT_PASSWORD_FILES]
    amazon-ebs.{{user build&#95;name}}:                         [-f FORKS] [-M MODULE_PATH] [--list-tasks]
    amazon-ebs.{{user build&#95;name}}:                         [--list-tags] [--step] [--start-at-task START_AT_TASK]
    amazon-ebs.{{user build&#95;name}}:                         playbook [playbook ...]
    amazon-ebs.{{user build&#95;name}}: ansible-playbook: error: argument --scp-extra-args: expected one argument
   ......
==> amazon-ebs.{{user build&#95;name}}: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs.{{user build&#95;name}}: Terminating the source AWS instance...

swan

2023-03-07 06:50:24

ansible error can be addressed by following:

 export ANSIBLE_SCP_EXTRA_ARGS="-O"  
 export ANSIBLE_SSH_ARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa"

Yike Wang

2023-03-07 07:04:13

I re-read the errors on master, it seems it is just the workaround of export ANSIBLESCPEXTRAARGS="-O" brought to the error:

Executing Ansible: ansible-playbook -e packer_build_name="flatcar-stable" -e packer_builder_type=amazon-ebs --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url=

ansible-playbook: error: argument --scp-extra-args: expected one argument

So I remove the env ANSIBLESCPEXTRAARGS and it can continue now.

👍 swan

Yike Wang

2023-03-07 07:08:59

Still the same thing fo build-ami-flatcar currently on master/0.1.14:

    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Enable kubelet service] *
    amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service kubelet: host"}
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: PLAY RECAP ***
    amazon-ebs.{{user build&#95;name}}: default                    : ok=49   changed=36   unreachable=0    failed=1    skipped=210  rescued=0    ignored=0

Jeremi Piotrowski

2023-03-07 09:25:25

whats in your config-1.26.2-flatcar.json

Jeremi Piotrowski

2023-03-07 09:25:40

so from your log, you're missing:

Jeremi Piotrowski

2023-03-07 09:25:43

when: kubernetes_source_type == "http" and kubernetes_cni_source_type == "http"

Jeremi Piotrowski

2023-03-07 09:25:49

these are set by default for AMIs

Jeremi Piotrowski

2023-03-07 09:25:53

flatcar AMIs

Jeremi Piotrowski

2023-03-07 09:25:55

but not for you somehow

Yike Wang

2023-03-07 10:39:52

i see! Let me figure out! Thanks!!!

Jeremi Piotrowski

2023-03-07 10:56:46

if you just care about getting an image this should work too:

diff --git a/images/capi/ansible/roles/kubernetes/tasks/main.yml b/images/capi/ansible/roles/kubernetes/tasks/main.yml
index 36d973b39..55885f1ef 100644
--- a/images/capi/ansible/roles/kubernetes/tasks/main.yml
+++ b/images/capi/ansible/roles/kubernetes/tasks/main.yml
@@ -21,6 +21,9 @@
 - import_tasks: photon.yml
   when: kubernetes_source_type == "pkg" and ansible_os_family == "VMware Photon OS"

+- import_tasks: url.yml
+  when: ansible_os_family == "Flatcar"
+
 - name: Symlink cri-tools
   file:
     src:   "/usr/local/bin/{{ item }}"

Yike Wang

2023-03-07 11:30:12

Thanks @Jeremi Piotrowski I added kubernetessourcetype == "http" and kubernetescnisource_type == "http" to my config json and then it works. I wasn't aware of these two things before. Thanks a lot!

Jeremi Piotrowski

2023-03-07 12:16:41

It shouldn’t be needed since it’s part of the flatcar config that is included in before your file

Jeremi Piotrowski

2023-03-07 12:16:51

But something about yours must be interfering

Yike Wang

2023-03-07 13:25:29

I just use this config json but I didn't add below two lines before (not aware of it ..):

cat config-1.26.2-flatcar.json
{
 "kubernetes_series": "1.26",
 "kubernetes_semver": "v1.26.2",
 "kubernetes_rpm_version": "1.26.2-0",
 "kubernetes_deb_version": "1.26.2-00",
 "kubernetes_source_type": "pkg",
 "kubernetes_http_source": "",
 "kubernetes_rpm_repo": "",
 "kubernetes_rpm_gpg_key": "\" \"",
 "kubernetes_rpm_gpg_check": "True",
 "kubernetes_deb_repo": "\" kubernetes-xenial\"",
 "kubernetes_deb_gpg_key": "",
 "kubernetes_container_registry": "registry.k8s.io",
 "kubernetes_load_additional_imgs": "false",
 "kubeadm_template": "etc/kubeadm.yml",
 "containerd_version": "1.6.8",
 "containerd_sha256": "8e227caa318faa136e4387ffd6f96baeaad5582d176202fe9da69cde87036033",
 #"kubernetes_source_type": "http",
 #"kubernetes_cni_source_type": "http"
}

Varun Kumar Vellanki

2023-03-06 17:48:58

@Varun Kumar Vellanki has joined the channel

Jhonathan Cavalcante

2023-03-06 22:27:31

[openstack Built CAPI Openstack Image breaking in kubeadm init command]

Jhonathan Cavalcante

2023-03-06 22:27:35

I'm building a kubernetes image for openstack. The build processes is running ok, the image is built and I'm able to provision a instance with it. The problem is that apparently cloud-init is running a kubeadm init command with the wrong kubeadm.yaml config, it looks like it is over righting the config with a file in /run/kubeadm/kubeadm.yaml. This is causing coredns pull to fail and break the provisioning process.

I've changed the registry value in the following files:

cloudinit/user-data:      imageRepository: registry.k8s.io
packer/config/kubernetes.json:  "kubernetes_container_registry": "registry.k8s.io",
packer/qemu/packer.json:    "kubernetes_container_registry": "registry.k8s.io",
packer_kubernetes.json:  "kubernetes_container_registry": "registry.k8s.io",

Not sure how cloud-init works on spining up the vms when I ask for a new instance, maybe there's some init bootstrap config to be made.

I've started to have this issue with k8s 1.24.10, the image built works well with k8s 1.22.9. I've also noticed some issues around the k8s registry, but nothing about this particular issue while provisioning the instance. I'd be glad if anyone can help me with this. Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-07 08:00:27

Which version of kube-builder are you using? This should have been resolved with

GitHub

fix: updated image registry to new domain by AverageMarcus · Pull Request #956 · kubernetes-sigs/image-builder

Signed-off-by: Marcus Noble <a href="mailto:github@marcusnoble.co.uk">github@marcusnoble.co.uk</a> What this PR does / why we need it: Switched the default container image registry to the new <a href="http://registry.k8s.io">registry.k8s.io</a>. The new Kubernetes v1.25.0 release...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/956

Jhonathan Cavalcante

2023-03-07 13:23:35

I'm using the latest release, just check now there's some tags with other releases, I'll be updating with all the changes. But we already had this configs inside the repo.

The thing is that cloud-init is apparently running a kubeadm init command that overrides this configuration from user-data.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-07 13:27:52

Is it that the kubeadm version being used is too old? Which provider is used for Openstack in image-builder?

Jhonathan Cavalcante

2023-03-07 17:43:37

I'm using qemu in ubuntu 20.04

Jhonathan Cavalcante

2023-03-07 17:55:49

just built from the last tag 0.1.14, and got the same error. it seems that kubeadm is writing its own kubeadm config file

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-09 12:38:49

Sorry, just got chance to look at this. It seems that the containerd_version isn't specified for that target so it relies on whatever is installed from Ubuntu 20.04 I think. I guess we need to specify the specific version to use. I'm not sure what version exactly is best.

Jhonathan Cavalcante

2023-03-09 12:40:57

Some updates on this. CAPI was overwriting the container registry address, so It was a issue on my side. But the 1.26.1 version of crictl is breaking a 1.22.9 k8s image for example. I think that can be related to containerd_version.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-09 12:43:03

Possibly, I get quite confused with all the inter-dependancies 😅

Jhonathan Cavalcante

2023-03-09 12:43:24

It is quite complex to understand

Sarva Bhowma

2023-03-08 20:56:08

@Sarva Bhowma has joined the channel

Abhay Krishna Arunachalam

2023-03-09 19:43:03

Hello Image-builder maintainers, I have a question. The packer.json for AMI builds has a throughput field by default in the block device mappings section with a default value of 125, but this field is valid only for gp3 volumes. I tried building a gp2 Ubuntu AMI by overriding the throughput from a different JSON var-file by setting it to "" or even null , but in both cases the build failed saying

Throughput is not available for device /dev/sda1

I was able to build it only after removing the throughput field from the packer.json. But removal also doesnt seem to be a solution because now there's no way of passing throughput in if I do want to build a GP3 volume. AMI missing something (pun intended 😛)? Is this a bug?

Abhay Krishna Arunachalam

2023-03-10 04:39:13

Created an issue for this:

GitHub

Unable to override throughput requirement when building gp2 AMI builds · Issue #1102 · kubernetes-sigs/image-builder

The packer.json for AMI builds has a throughput field in the block device mappings section with a default value of 125, but this field is valid only for gp3 volumes. I tried building an Ubuntu gp2 ...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1102

👍 jsturtevant

Sanika Gawhane

2023-03-10 22:23:21

Hello everyone, @kiran keshavamurthy and I are trying to bump the upstream image-builder to the latest tag/release v0.1.14 . Kiran created the tag and release successfully. However, the container build that was automatically kicked off pushed the container to the registry without a tag.
Are we missing something here? Does anything in cloudbuild.yaml need to be modified while creating the tag/release?

GitHub

image-builder/cloudbuild.yaml at master · kubernetes-sigs/image-builder

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/master/cloudbuild.yaml

Abhay Krishna Arunachalam

2023-03-13 21:16:27

@kiran keshavamurthy what do you think of these changes?
Background: We were hit by a Packer build issue where the incorrect packer binary was being used. We were building on a RHEL image, which already had a system-level package providing a packer binary, which conflicted with Hashicorp's Packer

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-14 15:51:13

Hey y'all 👋 are any of the other contributors and maintainers of image-builder going to be at Kubecon next month in Amsterdam? I'd be interested in a meetup to discuss the project, pain points, future, etc. if anyone was keen. Maybe at the contributor summit? (/cc @kiran keshavamurthy @jsturtevant @mboersma)

jsturtevant

2023-03-14 16:16:15

I won't be there this time

😞 Marcus Noble

mboersma

2023-03-14 19:41:39

I won't either, but I very much endorse having a summit meeting about image-builder and establishing some future directions, etc.

💙 Marcus Noble, Julien Klaer

kiran keshavamurthy

2023-03-16 19:07:46

I won’t be there either.

kiran keshavamurthy

2023-03-16 19:08:03

I think a restart of the office hours would be useful.

👍 mboersma, knfoo, jsturtevant, Marcus Noble

jsturtevant

2023-03-17 15:24:32

agreed, we have had quite a bit of activity in the last few months.

:nod: Marcus Noble

jsturtevant

2023-03-17 15:49:05

In other sigs we used a doodle to collect times, or we can set one that works for the majority of the maintainers. thoughts?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-17 16:01:08

I think a doodle would be useful. I think we have quite a split between US and EU timezones so I think it might be hard to find a time that pleases everyone.

👍 jsturtevant

jsturtevant

2023-03-17 16:16:46

any interest in taking lead on that?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-17 16:23:05

Sure, but I'll pick it up on Monday. 🙂

:ty: jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-20 09:23:09

Wow it's been a long time since I've used Doodle. It's got so many ads these days 🙈

Anyway, before I share into the main channel I'd like to get your thoughts first so here it is:

I went with constraints of no early than 8am PDT and no later than 7pm GMT. Hopefully that's good enough for most people.

doodle.com

Doodle

Doodle is the simplest way to schedule meetings with clients, colleagues, or friends. Find the best time for one-to-ones and team meetings with our user-friendly calendar tool. Get started today!

Original URL: https://doodle.com/meeting/participate/id/e165o8Pa

👍 jsturtevant

jsturtevant

2023-03-20 16:03:27

@kiran keshavamurthy @mboersma ^^^

kiran keshavamurthy

2023-03-20 17:16:55

Thanks for setting it up Marcus. I’ve submitted my times.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-20 17:19:59

Cool. I'll post it into the channel then. I'm thinking deadline for the end of this month to give people some time to answer but not have it drag on too long.

jsturtevant

2023-03-20 18:11:31

The days that have times today are grey out since they are in the past 🤨. Might need to put the dates in April

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-20 18:28:59

🤦‍♂️ I'll try and update it. It might cause the answers to need re-doing.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-20 18:29:56

Actually, I'm just going to create a new one and update the link. I'll let you know when its up so you can answer again. Sorry about that.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-20 18:35:08

Right, updated the link to
@jsturtevant @kiran keshavamurthy you'll need to fill it out again.

:done_2: kiran keshavamurthy, jsturtevant, mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-28 08:52:44

Just a reminder that we're looking for peoples thoughts on the best day/time to run the office hours. Could all those who might be interested in attending at some point please fill out the above Doodle poll with your favoured slot. 🙏

👍 mboersma, jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-03-30 15:19:04

Just one last reminder asking for peoples thoughts on this. 🙂 I'm going to close the poll tomorrow afternoon (UTC).
So far, Monday at 3:30pm UTC looks to be the favoured new slot for the office hours.

👍 jsturtevant, mboersma, kiran keshavamurthy

pengzhan

2023-03-21 03:45:52

@pengzhan has joined the channel

Abhishek Jha

2023-03-22 02:07:00

@Abhishek Jha has joined the channel

Maximilian Rink

2023-03-23 18:07:46

hey folks,
we are running into some issues with the ubuntu2204 images, namely with cloud-init. Maybe someone has seen it before 😅

Maximilian Rink

2023-03-23 18:08:17

That error isnt present at every start, but on about 70% of them

image.png

Drew Hudson-Viles

2023-03-27 14:05:07

Can you log into the node and check the cloud-init logs in /var/log?

I'm curious to know what you're seeing in there as I've ran into an issue in the past few days since I've synced with upstream and it looks like cloud-init might be failing. It may be purely coincidental though and I'm not sure if it's a problem with image builder as I've tested using an image that was built before I started having this issue.
It was previously working and is currently not making me wonder if this could be a problem with CAPI itself. I'm still looking into this on my side.

Drew Hudson-Viles

2023-03-28 16:52:21

I forgot to update you here. It was just a configuration issue on my side causing the cloud-init issue picard_facepalm. So it's very likely I can't assist you with my fix. However I do recommend if it's failing to check the cloud init logs to see what's going on in there.

aniruddha

2023-03-27 17:08:33

@aniruddha has joined the channel

aniruddha

2023-03-27 17:09:02

Need some final review on this

cc @richcase @kiran keshavamurthy

Abhay Krishna Arunachalam

2023-03-29 00:25:38

@kiran keshavamurthy Thoughts on this?

This is a duplicate of another PR that i had opened earlier and closed (#870)

GitHub

Add support for RHEL 8 RAW image builds by abhay-krishna · Pull Request #1120 · kubernetes-sigs/image-builder

Adding support for RHEL 8 RAW image builds.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1120

knfoo

2023-03-29 15:13:31

Hi 👋
I am preparing a PR on ubuntu-22.04 efi on qemu which requires a user-data file that is different from the file that is in ova - can you advise on where to put that file / name it differently like prefix it with qemu-

knfoo

2023-03-29 15:16:55

Or should this question be discussed in the PR ?

Drew Hudson-Viles

2023-03-29 20:05:15

Hello!

I'd say stick the PR in as a draft and then have the discussion in there. It'll give a "paper trail" for anyone else who may need to do something similar in future and helps any reviewers make decisions based on the conversation that happens in there. I recently had a pretty large PR go in that had something like 94 entries in the conversation tab (minus a few bot entries) but it was great to have it in there as anyone coming into the conversation had the history.

knfoo

2023-03-30 20:03:49

Great I will do that 👍

knfoo

2023-03-31 05:28:56

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-03 07:13:42

Right, closing the above poll. Based on the votes it looks like Mondays at 3:30pm UTC is the most favoured time slot for the office hours!

🎉 bavarianbidi, Drew Hudson-Viles, mboersma, jsturtevant

Drew Hudson-Viles

2023-04-04 12:15:29

Do you have a date for when the the first one will take place? (I only ask as the link takes me to March 30th 😉 )

Will you also drop a link in here for access to the meeting? It's going to be a first for me so not sure where to go for any of it 🙂

Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-04 12:18:27

Unsure right now. We wanted to find a suitable slot first.

Drew Hudson-Viles

2023-04-04 13:36:28

No problem 🙂 I'm getting ahead of myself!

mboersma

2023-04-05 17:49:15

Next Monday, April 10? I can be there.

👍 Drew Hudson-Viles, jsturtevant

jsturtevant

2023-04-05 21:43:42

We can get the readme updated: .

GitHub

GitHub - kubernetes-sigs/image-builder: Cross provider Kubernetes image building utility.

Cross provider Kubernetes image building utility. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder#community-discussion-contribution-and-support

jsturtevant

2023-04-05 21:44:18

I not sure who has access to update the calendar and zoom meeting time? @kiran keshavamurthy?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-06 09:33:36

The next two Mondays are no good for me (UK bank holiday then Kubecon) but would be happy for it to go ahead without me 🙂

Drew Hudson-Viles

2023-04-06 09:37:46

oh that's a good point - both of those things are applicable to me too. Baby brain is a killer.

😅 Marcus Noble

mboersma

2023-04-06 16:10:49

We can always wait a week or two if that's better timing. First we should sort out the calendar so the new time is publicly visible.

mboersma

2023-04-06 16:27:43

I created the kubernetes/community PR to change the meeting time here:
(However, when I changed the time previously for CAPZ office hours, this wasn't sufficient to actually make the change on the public calendar, we may have to nudge someone...)

GitHub

sig-cluster-lifecycle: update image-builder meeting to Mondays by mboersma · Pull Request #7245 · kubernetes/community

Updates the time of image-builder meetings to Mondays at 8:30 am PT (3:30 pm UTC), based on consensus from the community. See <https://kubernetes.slack.com/archives/C01E0Q35A8J/p1680185944828349?thr>...

Original URL: https://github.com/kubernetes/community/pull/7245

✅ Marcus Noble

:thx_thanks: Marcus Noble, kiran keshavamurthy

jsturtevant

2023-04-19 19:23:22

I've updated here too:

GitHub

Update time and day for community meeting by jsturtevant · Pull Request #1138 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1138

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-24 12:50:37

How do we go about getting this added to the public calendar?

mboersma

2023-04-24 16:19:14

I’m not sure. I thought the kubernetes/community PR would do it.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-24 16:21:31

Doesn't look like it I don't think. Unless my GCal isn't updating it? thinking

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-24 16:23:43

Yeah, not showing there:

Google Workspace

Google Calendar - Easier Time Management, Appointments & Scheduling

Learn how Google Calendar helps you stay on top of your plans - at home, at work and everywhere in between.

Original URL: https://calendar.google.com/calendar/u/0/embed?src=calendar@kubernetes.io&ctz=America/Los_Angeles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-24 16:26:19

This seems to suggest a specific calendar needs to be created and shared. Not sure if that's still correct or not -

Kubernetes Contributors

Calendar and Meeting Guidelines

Calendar and meeting guidelines, best practices, tips and troubleshooting guidance.

Original URL: https://www.kubernetes.dev/docs/comms/calendars/

👍 mboersma

Nitish Chauhan

2023-04-04 05:50:33

@Nitish Chauhan has joined the channel

Praveen Adini

2023-04-17 21:48:25

@Praveen Adini has joined the channel

sirhopcount

2023-04-18 09:02:49

@sirhopcount has joined the channel

Praveen Adini

2023-04-18 15:46:48

Hi Team, I built a RHEL 8 worker node AMI using image-builder and when I manually created a nodegroup for an existing EKS using an ASG, the nodes didn't get attached to the cluster. Upon further investigation I see some of the necessary scripts are missing from the image like /etc/eks/bootstrap.sh, etc. Wondering if its possible to build a worker node AMI using image-builder and if so, are they any further customizations that I have to perform.

voor

2023-04-18 20:11:22

Historically image builder was designed for kubeadm based clusters, of which EKS (to the best of my knowledge) is not one, I would recommend looking at

GitHub

GitHub - awslabs/amazon-eks-ami: Packer configuration for building a custom EKS AMI

Packer configuration for building a custom EKS AMI - GitHub - awslabs/amazon-eks-ami: Packer configuration for building a custom EKS AMI

Original URL: https://github.com/awslabs/amazon-eks-ami

Praveen Adini

2023-04-18 20:24:18

thanks @voor

rodrigodelmonte

2023-04-20 10:21:02

@rodrigodelmonte has joined the channel

2023-04-20 14:32:38

Hi, we build our capz flatcar images from the flatcar community library as a base but the 3510.2.0 is not available yet

pinging @Mateusz Gozdek (invidian) for reference ( since in the past you triggered the push )

➜ az sig image-version list-community --location westeurope --public-gallery-name flatcar-23485951-527a-48d6-9d11-6931ff0afc2e --only-show-errors --gallery-image-definition flatcar-stable-amd64 -o json | jq '.[]
.name'
"3374.2.0"
"3374.2.1"
"3374.2.3"
"3374.2.4"
"3374.2.5"

l4z41

2023-04-22 10:11:09

@l4z41 has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-24 15:57:45

Hey y'all 👋, the image-builder office hours will be taking place in roughly 30 minutes at half past.
You can find the meeting notes here:

🙌 Drew Hudson-Viles, mboersma, jsturtevant

jsturtevant

2023-04-24 19:59:31

Looking for some feedback on removing some of the projects which are not actively maintained:

:thx_thanks: Marcus Noble

👍 Drew Hudson-Viles

🧹 mboersma

Drew Hudson-Viles

2023-04-24 22:33:02

@Marcus Noble @jsturtevant @mboersma
Thought you may be interested in this based on the earlier discussion during office hours:

docs.nvidia.com

NVIDIA Virtual GPU Software License Server End of Life Notice (August 31, 2022) :: NVIDIA Virtual GPU Software News and Updates

This product change notice serves as formal notification that the NVIDIA virtual GPU software license server will reach End of Life (EOL) on July 31, 2023. After this date, NVIDIA virtual GPU software license server will have no new features, performance enhancements, bug fixes, or security updates. Technical support and software downloads for the NVIDIA virtual GPU software license server will no longer be available.

Original URL: https://docs.nvidia.com/grid/news/vgpu-software-license-server-eol-notice/index.html

👍 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-04-25 11:08:32

@mboersma I just noticed that the image-builder event is now correct in the Kubernetes calendar! 😄 Was that your doing?

mboersma

2023-04-25 17:36:45

@cecile is an SCL lead, she fixed it. 🙂

🎉 Marcus Noble

Mike Resvanis

2023-04-26 13:36:35

@Mike Resvanis has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-02 09:41:23

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 2:30PM every other Monday (next occurrence is May 8th), Greenwich Mean Time.

:ty: jsturtevant, mboersma

mloskot

2023-05-04 09:58:39

@mloskot has joined the channel

mloskot

2023-05-04 10:02:56

Hi, I'm new here, so hello everyone. As part of my activities around the sig-windows-dev-tools, I'm happy to share that after several days of attempts I've completed my first successful, I think, run of the image-builder on Windows generating Windows Server image running the builder from WSL. I do realise WSL is not a typical environment for image-builder users, but I just wanted to share that it is feasible to use WSL.

Mateusz Loskot (https://kubernetes.slack.com/team/U04R6C6HHC5)

Success! @jayunit100-pub-chnl-plz I think I've successfully built Windows Server 2019 image running the <code>image-builder</code> on WSL with Ansible+Packer executing against VirtualBox on Windows host. Here is what it generated after 1h 50m of run time: <code>$ ls -la image-builder/images/capi/output/windows-2019-kube-v1.24.11/win** -rwxrwxrwx 1 mloskot mloskot 4713108480 May 3 21:17 image-builder/images/capi/output/windows-2019-kube-v1.24.11/windows-2019-kube-v1.24.11-disk001.vmdk -rwxrwxrwx 1 mloskot mloskot 8078 May 3 21:10 image-builder/images/capi/output/windows-2019-kube-v1.24.11/windows-2019-kube-v1.24.11.ovf</code> Full log, <code>README_WSL.md</code> and scripts changes in my branch <a href="https://github.com/mloskot/sig-windows-dev-tools/tree/ml/master-image-builder-with-wsl">https://github.com/mloskot/sig-windows-dev-tools/tree/ml/master-image-builder-with-wsl</a>

🎉 jsturtevant

Christophe Jauffret

2023-05-04 13:27:25

@Christophe Jauffret has joined the channel

Yike Wang

2023-05-05 11:14:37

Hi folks, when I use v0.1.14 tag branch to build ami, I meet an issue:

amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": true, "cmd": "kubeadm config images pull --config /etc/kubeadm.yml --cri-socket /var/run/containerd/containerd.sock", "delta": "0:00:00.036572", "end": "2023-05-05 03:06:53.596128", "msg": "non-zero return code", "rc": 1, "start": "2023-05-05 03:06:53.559556", "stderr": "your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}

any idea? Thanks!

Yike Wang

2023-05-05 11:16:24

Full log:

no_proxy=** make build-ami-ubuntu-2004
hack/ensure-ansible.sh
Starting galaxy collection install process
Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using --force.
hack/ensure-ansible-windows.sh
IMPORTANT: Winrm connection plugin for Ansible on MacOS causes connection issues.
See  for more details.
To fix the issue provide the enviroment variable 'no_proxy='
Example call to build Windows images on MacOS: 'no_proxy= make build-'
hack/ensure-packer.sh
hack/ensure-goss.sh
Right version of binary present
packer build -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/cni.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/containerd.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/wasm-shims.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/common.json"  -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/additional_components.json"  -color=true -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/ami/ubuntu-2004.json" -var-file="/Users/yikew/Working/capa/image-builder/1.27/config.json"  packer/ami/packer.json
amazon-ebs.{{user build&#95;name}}: output will be in this color.

Yike Wang

2023-05-05 11:16:42

==> amazon-ebs.{{user `build_name`}}: Prevalidating any provided VPC information
==> amazon-ebs.{{user `build_name`}}: Prevalidating AMI Name: capa-ami-ubuntu-20.04-v1.27.0-1683255429
    amazon-ebs.{{user `build_name`}}: Found Image ID: ami-0481e8ba7f486bd99
==> amazon-ebs.{{user `build_name`}}: Creating temporary keypair: packer_64547086-412f-e577-58a5-b431924ffd0d
==> amazon-ebs.{{user `build_name`}}: Creating temporary security group for this instance: packer_6454708e-4b3f-2bec-e059-ec5d99783a1b
==> amazon-ebs.{{user `build_name`}}: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs.{{user `build_name`}}: Launching a source AWS instance...
    amazon-ebs.{{user `build_name`}}: Instance ID: i-00e09efbb79654a7b
==> amazon-ebs.{{user `build_name`}}: Waiting for instance (i-00e09efbb79654a7b) to become ready...
==> amazon-ebs.{{user `build_name`}}: Using SSH communicator to connect: 44.202.183.215
==> amazon-ebs.{{user `build_name`}}: Waiting for SSH to become available...
==> amazon-ebs.{{user `build_name`}}: Connected to SSH!
==> amazon-ebs.{{user `build_name`}}: Provisioning with shell script: /var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/packer-shell604033847
==> amazon-ebs.{{user `build_name`}}: Provisioning with shell script: ./packer/files/flatcar/scripts/bootstrap-flatcar.sh
==> amazon-ebs.{{user `build_name`}}: Provisioning with Ansible...
    amazon-ebs.{{user `build_name`}}: Setting up proxy adapter for Ansible....
==> amazon-ebs.{{user `build_name`}}: Executing Ansible: ansible-playbook -e packer_build_name="ubuntu-20.04" -e packer_builder_type=amazon-ebs --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=1d86b534c7bba51b78a7eeb1b67dd2ac6c0edeb01c034cc5f590d5ccd824b416 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.6.20 containerd_wasm_shims_url= containerd_wasm_shims_version=v0.3.3 containerd_wasm_shims_sha256=da84b1c065a58f95a841d39e143cd7115d43e6faedcce7a8782f2942388260d7 containerd_wasm_shims_runtimes="" crictl_url= crictl_sha256= crictl_source_type=pkg custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_http_source= kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key=" " kubernetes_rpm_gpg_check=True kubernetes_deb_repo=" kubernetes-xenial" kubernetes_deb_gpg_key= kubernetes_cni_deb_version=1.2.0-00 kubernetes_cni_rpm_version=1.2.0-0 kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=pkg kubernetes_semver=v1.27.0 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.27.0-00 kubernetes_rpm_version=1.27.0-0 no_proxy= pip_conf_file= python_path= redhat_epel_rpm= epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= build_target=virt amazon_ssm_agent_rpm= --extra-vars  --extra-vars  --scp-extra-args "-O" -e ansible_ssh_private_key_file=/var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/ansible-key68078117 -i /var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/packer-provisioner-ansible1682039367 /Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/ansible/node.yml
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: PLAY [all] ******************************************************************************************************************************************
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [Gathering Facts] ******************************************************************************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [include_role : node] **********************************************************************************************************
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [setup : Put templated sources.list in place] **********************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [setup : Put templated apt.conf.d/90proxy in place when defined] ********************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [setup : perform a dist-upgrade] ************************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [setup : install baseline dependencies] **********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [setup : install extra debs] ********************************************************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [setup : install pinned debs] ******************************************************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Ensure overlay module is present] ******************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Ensure br_netfilter module is present] ********************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Persist required kernel modules] ********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Set and persist kernel params] ************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.bridge.bridge-nf-call-iptables', 'val': 1})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.bridge.bridge-nf-call-ip6tables', 'val': 1})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv4.ip_forward', 'val': 1})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv6.conf.all.forwarding', 'val': 1})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv6.conf.all.disable_ipv6', 'val': 0})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv4.tcp_congestion_control', 'val': 'bbr'})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'vm.overcommit_memory', 'val': 1})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'kernel.panic', 'val': 10})
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'kernel.panic_on_oops', 'val': 1})

Yike Wang

2023-05-05 11:16:58

amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Ensure auditd is running and comes on at reboot] ************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : configure auditd rules for containerd] ********************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Ensure reverse packet filtering is set as strict] **********************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Copy udev etcd network tuning rules] ************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [node : Copy etcd network tuning script] ********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [include_role : providers] ************************************************************************************************
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : include_tasks] **********************************************************************************************
    amazon-ebs.{{user `build_name`}}: included: /Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/ansible/roles/providers/tasks/aws.yml for default
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : upgrade pip to latest] ******************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : install aws clients] **********************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : install aws agents Ubuntu] **********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Ensure ssm agent is running Ubuntu] ****************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Disable Hyper-V KVP protocol daemon on Ubuntu] ******************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Creates unit file directory for cloud-final] **********************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Create cloud-final boot order drop in file] ************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Creates unit file directory for cloud-config] ********************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Create cloud-final boot order drop in file] ************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Make sure all cloud init services are enabled] ******************************
    amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-final)
    amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-config)
    amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-init)
    amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-init-local)
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Create cloud-init config file] **************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : set cloudinit feature flags] ******************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [providers : Ensure chrony is running] ************************************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [include_role : containerd] **********************************************************************************************
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Install libseccomp2 package] ****************************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : download containerd] ********************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Create a directory if it does not exist] ****************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : unpack containerd] ************************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : delete /opt/cni directory] ********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : delete /etc/cni directory] ********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Creates unit file directory] ****************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Create containerd memory pressure drop in file] **************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Create containerd max tasks drop in file] **************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Create containerd http proxy conf file if needed] **********************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Creates containerd config directory] ************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Copy in containerd config file etc/containerd/config.toml] ******
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : Copy in crictl config] ****************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : start containerd service] **********************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : delete tarball] ******************************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [containerd : delete tarball] ******************************************************************************************
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [include_role : kubernetes] **********************************************************************************************
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Add the Kubernetes repo key] ****************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Add the Kubernetes repo] ************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Install Kubernetes] **********************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Symlink cri-tools] ************************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item=ctr)
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item=crictl)
    amazon-ebs.{{user `build_name`}}: changed: [default] => (item=critest)
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Create kubelet default config file] **************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Enable kubelet service] **************************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Create the Kubernetes version file] **************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Check if Kubernetes container registry is using Amazon ECR] ******
    amazon-ebs.{{user `build_name`}}: ok: [default]
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Create kubeadm config file] ******************************************************************
    amazon-ebs.{{user `build_name`}}: changed: [default]
    amazon-ebs.{{user `build_name`}}:

Yike Wang

2023-05-05 11:17:03

amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Kubeadm pull images] ********************************************************************************
    amazon-ebs.{{user `build_name`}}: fatal: [default]: FAILED! => {"changed": true, "cmd": "kubeadm config images pull --config /etc/kubeadm.yml --cri-socket /var/run/containerd/containerd.sock", "delta": "0:00:00.036572", "end": "2023-05-05 03:06:53.596128", "msg": "non-zero return code", "rc": 1, "start": "2023-05-05 03:06:53.559556", "stderr": "your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
    amazon-ebs.{{user `build_name`}}:
    amazon-ebs.{{user `build_name`}}: PLAY RECAP ******************************************************************************************************************************************
    amazon-ebs.{{user `build_name`}}: default                    : ok=55   changed=43   unreachable=0    failed=1    skipped=205  rescued=0    ignored=0
    amazon-ebs.{{user `build_name`}}:
==> amazon-ebs.{{user `build_name`}}: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs.{{user `build_name`}}: Terminating the source AWS instance...
==> amazon-ebs.{{user `build_name`}}: Cleaning up any extra volumes...
==> amazon-ebs.{{user `build_name`}}: No volumes to clean up, skipping
==> amazon-ebs.{{user `build_name`}}: Deleting temporary security group...
==> amazon-ebs.{{user `build_name`}}: Deleting temporary keypair...
Build 'amazon-ebs.{{user `build_name`}}' errored after 10 minutes 36 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 10 minutes 36 seconds

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs.{{user `build_name`}}: Error executing Ansible: Non-zero exit status: exit status 2

==> Builds finished but no artifacts were created.

Yike Wang

2023-05-05 11:17:53

my config file:

{
 "kubernetes_series": "1.27",
 "kubernetes_semver": "v1.27.0",
 "kubernetes_rpm_version": "1.27.0-0",
 "kubernetes_deb_version": "1.27.0-00",
 "kubernetes_source_type": "pkg",
 "kubernetes_http_source": "",
 "kubernetes_rpm_repo": "",
 "kubernetes_rpm_gpg_key": "\" \"",
 "kubernetes_rpm_gpg_check": "True",
 "kubernetes_deb_repo": "\" kubernetes-xenial\"",
 "kubernetes_deb_gpg_key": "",
 "kubernetes_container_registry": "registry.k8s.io",
 "kubernetes_load_additional_imgs": "false",
 "kubeadm_template": "etc/kubeadm.yml",
 "containerd_version": "1.6.20",
 "containerd_sha256": "1d86b534c7bba51b78a7eeb1b67dd2ac6c0edeb01c034cc5f590d5ccd824b416"
}

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-05 11:29:03

The error seems to suggest that you're using a now deprecated version of the kubeadm configfile.
This was resolved a couple weeks ago in but hasn't made it into a release yet. You'll be able to get around it for now by using the latest from the master branch until a new release is released.
@mboersma @jsturtevant we spoke about releases in the last office hours, looks like we could do with doing one so that we support Kubernetes v1.27 with a tagged release of image-builder.

GitHub

Update kubeadm config file to supported API version by mboersma · Pull Request #1133 · kubernetes-sigs/image-builder

What this PR does / why we need it: Updates the kubeadm.yml config file to the current v1beta3 API. Support for the v1beta2 API was dropped in Kubernetes v1.27.0, thus image-builder can't build cur...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1133

👏 Yike Wang

👍 Yike Wang

:thanks: Yike Wang

:1000000: Yike Wang

Yike Wang

2023-05-05 11:35:18

ok then let me use master branch to build 1.27 related amis. Thanks a lot!

jsturtevant

2023-05-11 21:29:35

we have a release out:

Matt Boersma (https://kubernetes.slack.com/team/U5PL62ULA)

Image-builder v0.1.15 is now available: <a href="https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.15">https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.15</a> Thanks to all the (many) contributors! :thank_you:

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1683835142044209?thread_ts=1683560653.140759&cid=C01E0Q35A8J

👏 Yike Wang

Slackbot

2023-05-08 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

Drew Hudson-Viles

2023-05-08 15:31:31

I may not make this office hours due to moving house and everything being all over the place in the new house right now. I will try though.

mboersma

2023-05-08 16:44:13

We need to do a fresh release of image-builder, but there is some confusion about the process and the last attempt apparently didn't publish a tagged image.

mboersma

2023-05-08 16:46:17

@kiran keshavamurthy @jsturtevant @Marcus Noble should we get together and see if we can sort things out? (Or does anyone already have a handle on things?)

:yes: Marcus Noble, jsturtevant

jsturtevant

2023-05-08 17:13:16

Yes, Sorry I missed the meeting today. I didn't have it on the calendar. I can do something later in the week?

👍 mboersma

kiran keshavamurthy

2023-05-08 23:46:41

Yeah I’m ok meeting sometime this week to figure it out.

mloskot

2023-05-09 13:11:50

confusion about the process

Is there anything in it related to the Windows images?

jsturtevant

2023-05-09 23:39:33

@mloskot not sure I follow your question. This for the release of the image-builder Docker image and tagging the repository. The docker image is Linux only but can get used with various providers to create Windows. Does that help?

jsturtevant

2023-05-09 23:40:32

folks up for doing a session on Thursday? Maybe 8:30 pacific so European timezone can join if they want?

👍 Marcus Noble, mboersma, kiran keshavamurthy

mloskot

2023-05-10 08:27:16

@jsturtevant Yes, I realise it's not very concrete. I was trying to make myself aware of image-builder issues w.r.t. Windows.

👍 jsturtevant

jsturtevant

2023-05-10 17:36:51

I don't know If I can create a meeting invite but we can re-use the link to the weekly meeting. We can all just sign on at that time.

mboersma

2023-05-10 17:56:29

Works for me, let's try for tomorrow a.m.

👍 jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-10 18:16:47

I’ll try and attend but might be late.

👍 jsturtevant

jsturtevant

2023-05-10 19:04:28

@kiran keshavamurthy does that work for you? I think you have the most context but I am sure we can figure it out 🙂

kiran keshavamurthy

2023-05-10 19:05:44

I think so. I have some home repair guys coming in between 8-10am. But I should be able to be on the call and chime in.

:ty: jsturtevant

mboersma

2023-05-11 16:29:41

Meeting to share knowledge about image-builder publishing process starting now at , please join if you're interested and available!

:ty: jsturtevant

✅ mloskot

jsturtevant

2023-05-11 16:30:18

I was just about to do this 🙂

jsturtevant

2023-05-11 17:28:22

We have a PR for promotion:

GitHub

Image promotion for scl-image-builder v0.1.15 by jsturtevant · Pull Request #5262 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.15 This is an automated PR generated from kpromo kpromo pr --fork jsturtevant --project scl-image-builder --reviewers "@jsturtevant @kkeshavamurthy @mboer...

Original URL: https://github.com/kubernetes/k8s.io/pull/5262

jsturtevant

2023-05-11 17:28:26

thanks all!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-11 17:28:59

Just found another location we need to update owners:

GitHub

k8s.io/OWNERS at main · kubernetes/k8s.io

Code and configuration to manage Kubernetes project infrastructure, including various *.<a href="http://k8s.io">k8s.io</a> sites - <a href="http://k8s.io/OWNERS">k8s.io/OWNERS</a> at main · kubernetes/k8s.io

Original URL: https://github.com/kubernetes/k8s.io/blob/main/registry.k8s.io/images/k8s-staging-scl-image-builder/OWNERS

👍 jsturtevant, mboersma

jsturtevant

2023-05-11 18:40:59

@kiran keshavamurthy We should get you added to the k8s org 🙂

GitHub

Add image-builder owners to OWNERS file by mboersma · Pull Request #5263 · kubernetes/k8s.io

Adds current image-builder maintainers to the relevant OWNERS file. cc: @jsturtevant @AverageMarcus @randomvariable

Original URL: https://github.com/kubernetes/k8s.io/pull/5263#issuecomment-1544331558

mboersma

2023-05-11 20:59:02

Image-builder v0.1.15 is now available:
Thanks to all the (many) contributors! thank_you

:nice: mloskot, Jon Zeolla

:wookie_party_time: Joe Kratzat

:parrotk8s: Marcus Noble

:yay2: knfoo

mboersma

2023-05-11 21:01:45

I've started a PR to add some Makefile stuff and docs for releasing, just FYI so we don't duplicate effort.

:ty: jsturtevant, Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-12 07:32:58

I just noticed that the next date in the office hours notes is down as the 15th but the next in the calendar is the 22nd. Am I correct that its a mistake in the notes or are we wanting to meet up early?

jsturtevant

2023-05-12 16:47:27

I believe the calendar is correct as we had one this week

mboersma

2023-05-12 16:53:06

That was my mistake, date math is hard. I updated the notes to the 22nd.

😃 jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-12 16:53:21

👍 Just wanted to be sure 🙂

mloskot

2023-05-13 11:08:34

Has anyone tried running the image builder with VirtualBox lately?
After pretty good experiments on Windows host (with WSL), I'm now trying the IB's canonical way, that is on Linux machine with Ubuntu 22.04 and I'm getting errors, but before I start spamming GitHub with issues and PRs, I'd like confirm if IB workflow w/ VirtualBox is still sounds or needs updating indeed.

jsturtevant

2023-05-16 17:30:42

are you building linux images on wsl? I haven't used VB provider from wsl

mloskot

2023-05-16 17:36:17

@jsturtevant
Yes, I have experimented with it, but it requires a hybrid environment: WSL is only used as proxy running Ansible, Packer and Vagrant, but VirtualBox runs on Windows host
Here is my branch with single commit with scratchnotes and changes I had to apply as well as full log

However, I've ditched this idea as there seem to be too much gymnastics needed, and hardcoded IPs, and I don't see how this could be made into generic changes approvable by image builder and SWDT.
Instead, I've got a Linux machine where I'm going to try building images

👍 jsturtevant

Xavier Serrat

2023-05-13 22:10:07

@Xavier Serrat has joined the channel

👋 jsturtevant

Daniel Mello Urbano

2023-05-16 13:20:59

@Daniel Mello Urbano has joined the channel

Danny Bessems

2023-05-22 12:57:18

I'm experiencing a weird error when building a RHEL8 ova for CAPV; after defining values in additional_components.json just like I've done for Ubuntu18/20/22:

  "additional_registry_images": "true",
  "additional_registry_images_list": "ghcr.io/kube-vip/kube-vip:v0.5.5",

I see the following error when Packer starts the Ansible provisioner:

    vsphere-iso.vsphere: TASK [Gathering Facts] **
    vsphere-iso.vsphere: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: command-line line 0: garbage at end of line; \"-o\".", "unreachable": true}

Danny Bessems

2023-05-22 13:24:54

Even if I use just node as image name (since it's shorter than ghcr.io/kube-vip/kube-vip:v0.5.5), I still get the same error

Danny Bessems

2023-05-22 14:51:27

Ok, figured it out; somehow packer/ansible/ssh in my build-environment image introduced this new behaviour; I rolled back that image and the issue is no longer present.

I'll try rebuilding with latest versions of all binaries involved and see if the behaviour re-occurs.

Danny Bessems

2023-05-22 15:34:15

Looks like the difference is Ansible 2.14.5 vs 2.15.0; pinning to 2.14.x to avoid the issue for now

👍 jsturtevant

jsturtevant

2023-05-22 18:00:27

can you open an issue about upgrading to ansible 2.15.0?

jsturtevant

2023-05-22 18:00:42

we will need to eventually upgrade.

jsturtevant

2023-05-22 18:00:44

Thanks!

Danny Bessems

2023-05-23 12:25:21

Sure thing, though I'm not using make deps to install stuff, so my environment is not necessarily representative.

👍 jsturtevant

Slackbot

2023-05-22 15:30:01

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

jsturtevant

2023-05-22 16:17:25

I'll be a few mins late

👍 Marcus Noble

Danny Bessems

2023-05-22 16:01:24

Something I've noticed between building node-templates (.ova's) for vSphere. Ubuntu 20.04 produced 1.7GB large .ova's, Ubuntu 22.04 produces 3.0-3.5GB large .ova's (for reference, RHEL8 is 1.8GB large).

What can explain the increased size between 20.04 and 22.04; should we add a custom role for 22.04 to uninstall a load of new packages (for instance the frustrating needrestart as just a random example)?

mboersma

2023-05-22 20:31:41

PSA: image-builder branch renaming and cruft removal

:megaman: jsturtevant

mboersma

2023-05-22 20:31:52

The image-builder project initially tried to collect VM image-building utilities for Kubernetes in one repository. Over time, the kubedeploy/imagebuilder and konfigadm tools have become unmaintained, while the images/capi area has remained active.

Users and developers are only interested in this latter "Image Builder for Cluster API" area. The presence of the other tools (which have not had code changes in over two years) is an impediment.

We (image-builder maintainers) propose removing these unmaintained projects to simplify the repository. Additionally, we would like to begin using "main" branch nomenclature at roughly the same time as the "cruft removal" described above.

If you have any feedback on these proposed changes, please let us know by commenting on either of the issues listed below before Tuesday, May 30.

Removing unmaintained projects from image-builder · Issue #1143 · kubernetes-sigs/image-builder (github.com )
Rename "master" branch to "main" · Issue #1161 · kubernetes-sigs/image-builder (github.com )

👍 cecile, Julien Klaer

Danny Bessems

2023-05-23 07:53:42

Hurray, then I can stop doing sparse checkouts on that repo 😄

😆 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-23 10:46:42

I have just learnt that test-infra has a component called "image-builder" 🙈

This might add some weight to the proposal of also renaming the project to be something like cluster-api-image-builder

GitHub

test-infra/README.md at master · kubernetes/test-infra

Test infrastructure for the Kubernetes project. Contribute to kubernetes/test-infra development by creating an account on GitHub.

Original URL: https://github.com/kubernetes/test-infra/blob/master/images/builder/README.md?plain=1

Danny Bessems

2023-05-23 10:49:12

In that train of thought, openshift has an imagebuilder too

🙈 Marcus Noble, mboersma

Danny Bessems

2023-05-23 10:49:59

It's unavoidable really; descriptive names for building images don't vary a lot 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2023-05-23 10:50:43

True. But if we can be more specific without it being a problem then that might be worth doing 🙂

Danny Bessems

2023-05-23 10:51:25

cluster-api- prefix makes perfect sense to me.

mikejoh

2023-05-23 16:04:55

@mikejoh has joined the channel

Ashutosh Upadhyay

2023-05-25 07:33:00

@Ashutosh Upadhyay has joined the channel

Danny Bessems

2023-05-25 12:47:36

Has anyone got experience with RHEL8 and timing issues with regards to containerd configuration (through - files)?

Danny Bessems (https://kubernetes.slack.com/team/UQU1LL28L)

I'm running into an interesting issue when using RHEL8 for my downstream clusters along with configuring containerd to use registry mirrors: In the <code>- files</code> list I have several <code>hosts.toml</code> files that get stored in their respective <code>/etc/containerd/certs.d/&lt;registry&gt;</code> folders. Unfortunately, it looks like containerd does not pick up the configuration without an explicit restart of containerd.service; when I compare that behaviour with Ubuntu2004/2204, the configuration gets picked up without issues. Is there a way to control the order of the cloud-init directives (write files before executing runcmds) from within kubeadmControlPlane and kubeadmConfigTemplate?

Shruthi P

2023-05-26 06:54:18

@Shruthi P has joined the channel

Shruthi P

2023-05-26 12:56:54

Hi Team, im trying to create ubuntu capi image. I would like to know how to authenticate it with root/capi user and password?

Drew Hudson-Viles

2023-05-26 12:59:47

You would need to add an SSH public key to it as part of your cloud-init as the password is not generally available. Once you have the key on the machine, you could log in using that and change the password as you wish. Personally I wouldn't recommend that due to the security implications that would introduce.

Shruthi P

2023-05-26 14:38:00

Thanks @Drew Hudson-Viles , pls tell me if this will work - in packer.json there is "ssh-username=builder" . So can i login like ssh builder@IP with password ? to create a image can i use PACKERFLAGS="--var 'kubernetesrpmversion=1.24.0-0' --var 'kubernetessemver=v1.24.0' --var 'kubernetesseries=v1.24' --var 'kubernetesdebversion=1.24.0-00' --var 'disksize=10240'" --var 'sshpassword=enggfusion' --var 'sshusername=enggfusion' make build-kubevirt-qemu-ubuntu-2004

Drew Hudson-Viles

2023-05-26 15:32:02

No problem.

So, the username and password bit you've referred to is for the builder, not the end result of an image - IE these will not be available on the image once you've built it.
See more information on the SSH communicator here.

If you're building from a standard, unaltered Ubuntu image then once the image is built, the only username available by default will be the one supplied by Ubuntu, which is ubuntu

To get your own username and password onto the image, you'd need to provide user-data via the cloud-init "method".

Depending on your infrastructure for creating the VM from this image this can vary.

Danny Bessems

2023-05-30 10:55:55

I've struggled with this exact issue quite a bit (if your nodes do not even get networking configured correctly for instance); my workaround in the end was to temporarily leave the builder account unlocked for debugging, and once issues were resolved replace it again with an image that has the builder account locked.

For reference; change the "shutdown_command" and remove usermod -L {{user ssh_username}} &&

Apricote

2023-05-27 19:21:53

@Apricote has left the channel

Fran

2023-05-31 08:54:17

@Fran has joined the channel

Fran

2023-05-31 09:58:03

Hi folks! I'm trying to get merged, is anyone free to take a look at it please?

GitHub

Add support for RHEL 8 to GCP capi provider image builder by kahun · Pull Request #1129 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adds support to build rhel 8 images to using with CAPG provider using cluster-api.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1129

janr

2023-05-31 10:12:12

@janr has joined the channel

Phani Mvs

2023-06-01 12:18:03

Any idea on who populates this instance-data.json file while provisioning using CAPV provider?
we are seeing issues with RHEL-8 vm templates as ds.meta_data.hostname is getting populated as ‘localhost’

cat instance-data.json 

{
 "base64_encoded_keys": [],
 "ds": {
 "meta_data": {
  "hostname": "localhost",
  "instance-id": "phani-rhel-26-1-9g5bt",
  "local-hostname": "phani-rhel-26-1-9g5bt",
  "local-ipv4": "10.109.10.99",
  "local-ipv6": "",
  "local_hostname": "localhost"
 }
}

Jeremi Piotrowski

2023-06-01 12:59:46

cloud-init generates that

👍 Phani Mvs

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-02 09:26:35

I've been spending some time working on the repo cleanup tasks today. I've got the following PRs opened to work towards the cleanup of old projects and the rename of master branch:

Cleanup unmaintained projects

Include the main branch alongside master in the Azure nightly job

Rename branch in Prow configs for other projects using image-builder (this one needs to be on hold until after the rename has been completed)

❤️ mboersma

:megaman: jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-02 09:27:44

I've also added the full checklist from to the issue to track what is outstanding with the rename -

Danny Bessems

2023-06-02 09:35:05

Will the folder images/capi move to the root of the repo afterwards?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-02 09:41:19

I think we'll leave it for now as not sure what kind of effect that'd then have on all the other projects relying on it.

Danny Bessems

2023-06-02 09:42:01

That makes sense yeah.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-02 09:43:04

Hopefully later we'll be able to move it but I think we should minimize the potential impact right now. 🙂

👍 jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-02 10:27:06

I'm not going to be able to make the office hours on Monday but I've added a couple notes to the existing items on the agenda. Feel free to ping me on Slack with anything needed 🙂

Mohd Waquar

2023-06-05 04:24:14

@Mohd Waquar has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 08:10:12

So... turns out that Packer like to introduce breaking changes into patch releases 🤨 We recently bumped Packed from v1.8.6 -> v1.8.7 and that release removed some vendor plugins, specifically for us DigitalOcean. 🤦‍♂️
I opened an issue to cover this:
And a PR to fix it by specifying the external plugin:

The original PR that did the version bump didn't seem to run the pull-packer-validate test that would have caught this. I'll be updating the test in test-infra to make sure this is triggered when changing the ensure script. -

👍 jsturtevant

Danny Bessems

2023-06-05 09:01:21

Hashicorp doesn't follow semantic versioning properly in all of their products 😞

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 09:02:42

Gotta keep folks on their toes it seems 😅

Drew Hudson-Viles

2023-06-05 09:09:02

So much just slid into place 🤣

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 09:10:42

@Drew Hudson-Viles you should be good to rebase your PR on master now and hopefully should all work 🤞

Drew Hudson-Viles

2023-06-05 09:11:54

Lovely. I'll get that done shortly then. Just fueling myself with coffee after a lovely night of a non sleeping baby.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 09:12:15

Ooof! Never fun.

mboersma

2023-06-05 16:17:21

D'oh, and that's why I didn't try to upgrade us to the 1.9.x series, where they mention removing default plugins. Seemed like 1.8.7 was safe, apologies!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 16:18:45

Nah it's ok. Lesson learned 🙂 I thought those tests would have ran on that PR you did so that should have caught it but they weren't actually configured to 🤦‍♂️

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 17:49:28

@jsturtevant Mind giving this an /approve ? You should be able to now we've had the owners updated. 🙂

GitHub

Trigger image-builder job when ensure-packer.sh is updated by AverageMarcus · Pull Request #29679 · kubernetes/test-infra

Related: kubernetes-sigs/image-builder#1180 The packer validate test needs to also trigger when we change the ensure-packer.sh script so we test changes related to Packer version updated. /assign @...

Original URL: https://github.com/kubernetes/test-infra/pull/29679

Slackbot

2023-06-05 15:30:12

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-05 17:33:51

Is there something up with the test infra? I think this PR is stuck -
Nevermind, sorted now it seems 🙂

feitnomore

2023-06-06 16:43:07

@feitnomore has joined the channel

feitnomore

2023-06-06 16:43:37

Hi... I'm working on building a few cluster images here, and I've got an issue with Rocky Linux 8, which was giving me a 404

feitnomore

2023-06-06 16:43:59

I've submitted a PR fixing the URL, as the URL was looking for 8.7 and the actual version is 8.8

👍 mboersma, Marcus Noble

:thx_thanks: Marcus Noble

feitnomore

2023-06-06 16:44:14

GitHub

Update Rocky Linux to 8.8 by weizhouapache · Pull Request #1184 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1184

feitnomore

2023-06-06 17:33:32

is there any roadmap to add new distros/versions? Like Ubuntu 23.04, Rocky Linux 9?

Christophe Jauffret

2023-06-06 18:54:27

Just open this issue as the commit
is breaking the loadadditionalcomponents feature

GitHub

couldn't resolve module/action 'amazon.aws.s3_object'. · Issue #1185 · kubernetes-sigs/image-builder

What steps did you take and what happened: since commit 5a25506 when we try to use load_additional_components it failed with the following error ERROR! couldn't resolve module/action 'amazon.aws.s3...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1185

Drew Hudson-Viles

2023-06-06 22:05:36

So, yes you're right, somehow the defaults are not there which is blowing my mind as I specifically remember putting them in originally. I can only presume I did something in the process of creating the addition that removed them. Probably an erroneous rebase picard_facepalm

I've got fix prepared for this that also adds them "back" in as part of it (though it's not the actual fix).

As for your notes on the aws ansible collection not being officially supported in the current version. It looks like I didn't hardcode the version and an update has occurred since the original PR was created. I'll hardcode the version in to prevent errors as 5.x.x supports 2.11.0+

I'm running a few tests against the fix now it now wrt the issue you've raised which includes me completely removing the collection to ensure it works without it installed, then I'll get a PR put in.

Christophe Jauffret

2023-06-07 07:16:09

perfect don’t hesitate to share me the PR for verification on my side

✅ Drew Hudson-Viles

Drew Hudson-Viles

2023-06-07 08:47:32

I'm just running one last test my side to ensure the Nvidia bits still work (It did last night but it was late and I want a non-tired brain to confirm it) and then we should be good to go.

GitHub

fix: s3 task causing errors with other tasks in load_additional_components by drew-viles · Pull Request #1186 · kubernetes-sigs/image-builder

What this PR does / why we need it: An error was occurring in the load_additional_components role wrt the s3 module not being available when people were using non-s3 tasks within it. This fix: Con...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1186

Christophe Jauffret

2023-06-07 08:54:28

PR seems OK, i just launch a test pipeline let’s see

Christophe Jauffret

2023-06-07 10:53:42

ny test pipeline is OK too

Drew Hudson-Viles

2023-06-07 10:54:00

\o/

Christophe Jauffret

2023-06-06 18:56:06

@Drew Hudson-Viles 👆

Drew Hudson-Viles

2023-06-06 18:57:43

Aaah thanks for raising this, I didn't hit this in my local testing but maybe didn't hit the use case of loadadditionalcomponents. I shall take a look into this.

Christophe Jauffret

2023-06-06 18:59:53

there is multiple issues here

lack of amazon.aws ansible collection

amazon.aws ansible collection is not officially supported with the default image-buidler ansible version 2.11.5

new roles values are missing in the images/capi/ansible/roles/loadadditionalcomponents/defaults/main.yml file

Drew Hudson-Viles

2023-06-06 19:06:49

It shouldn't really be affecting you if you're not using the role so I'll see what can be done about that. Ideally you should only need to include the hack file when making use of the S3 role/Nvidia role.

I'll take a look asap to get a fix in.

knfoo

2023-06-07 07:15:30

Hello folks 👋
I have this sitting - any feedback would be great 🙏

knfoo

2023-06-28 14:04:21

Any one ?

Joe Kratzat

2023-06-28 14:27:03

@knfoo I think since the PR is in draft …. it probably won’t get looked at

knfoo

2023-06-28 15:39:13

@Joe Kratzat OK, I was advised to put it into draft first - I can make it a real PR maybe it will get some attention then 🙂

Joe Kratzat

2023-06-28 15:48:33

oh don’t listen to me … I’m not a maintainer … 🤣
Just noticed it was in draft and that typically doesn’t ping people

knfoo

2023-06-28 15:48:58

🙂

mboersma

2023-06-29 21:11:16

Sorry @knfoo we clearly dropped the ball here. I took a look at it and don't see anything controversial.
It would be good to get more eyes on it, but at least one maintainer is on break and July 4 is imminent. Maybe if you promote it from draft status it will notify people?

knfoo

2023-06-30 07:04:02

@mboersma that is OK - I got attention now 🙂
I will promote it from draft and see what happens

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-08 08:25:27

Do we want to un-pin this issue from the GitHub issues page?

GitHub

Removing unmaintained projects from image-builder · Issue #1143 · kubernetes-sigs/image-builder

Is your feature request related to a problem? Please describe. I would like to propose removing the currently unmaintained projects and simplify this repository. The readme states there are three p...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1143

👍 mboersma, jsturtevant

jsturtevant

2023-06-08 16:36:41

done! thanks for the reminder

👍 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-09 09:51:16

I've just noticed we have another OWNERS file here that requests CAPI maintainers to give approvals. See example of confusion on this pr:

Anyone against removing this and just relying on the image-builder OWNERS file. Now we've only got the single project I don't think it makes sense to have both anymore.
/cc @mboersma @jsturtevant

👍 jsturtevant, mboersma, Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-12 11:00:22

PR -

GitHub

Remove leftover CAPI OWNERS file by AverageMarcus · Pull Request #1187 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1187

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-12 15:23:37

@mboersma Looks like it needs /lgtm also. I thought /approve implied that but I guess not 😕

Harshil Patel

2023-06-10 16:20:06

@Harshil Patel has joined the channel

Christophe Jauffret

2023-06-12 17:41:44

Hello team, is there a specific release cadency in place for image-builder ? if not what is the rules to trigger a new release ?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-12 17:57:14

It’s pretty ad-hoc right now.
Are you looking for a new release for a specific feature?

Christophe Jauffret

2023-06-12 17:58:41

yes
the EKS image-builder need to consume some change we put in the nutanix packer flavor
and he can only link to a specific tag or a release no more a commit
so we need a new tag or release 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-12 19:17:05

Makes sense.
@mboersma @jsturtevant I think we said recently we wanted to do a release before the branch rename anyway, right?

👍:skin_tone_2: Christophe Jauffret

jsturtevant

2023-06-12 20:45:31

I think that makes sense

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-13 08:27:17

I think one of y'all are going to need to do the release. I'm still waiting on this PR before I have the GitHub perms to create releases 😞

GitHub

Add AverageMarcus to image-builder-admins by mboersma · Pull Request #4262 · kubernetes/org

Syncs up the GH ownership with image-builder's OWNERS_ALIASES

Original URL: https://github.com/kubernetes/org/pull/4262

Christophe Jauffret

2023-06-13 08:36:24

no perm on my side too 🤷‍♂️

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-13 08:47:26

Sorry, was referring to Matt and James 🙂 They're US based so should be online later. I also need to chase up getting that PR approved so I have the right permissions too.

😅 Christophe Jauffret

mboersma

2023-06-13 18:22:31

I can do a release today, I'll get started in a bit. Thanks for the nudge @Christophe Jauffret.

Christophe Jauffret

2023-06-13 18:27:15

thanks @mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-13 18:41:49

Need any help @mboersma or you ok to handle it all? 🙂

mboersma

2023-06-13 18:43:10

Thanks, I've got it. I would pair with you @Marcus Noble but I'm in a sig-docs meeting and multitasking. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-13 18:54:29

No worries 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-13 18:54:34

Thanks for taking care of it 🙂

mboersma

2023-06-13 18:59:08

Ilya Alekseyev (NTNX)

2023-06-12 17:43:40

@Ilya Alekseyev (NTNX) has joined the channel

Sid

2023-06-12 21:38:18

@Sid has joined the channel

mboersma

2023-06-13 19:20:17

Image-builder v0.1.16 is now available:
Thanks to all the (many) contributors! thank_you

🎉 jsturtevant, Marcus Noble, Christophe Jauffret, Mitchel Haring, Julien Klaer, Fran, Drew Hudson-Viles, Ilya Alekseyev (NTNX), cecile

:kubernetes_intensifies: Julien Klaer

hislacker

2023-06-15 13:47:35

@hislacker has joined the channel

hislacker

2023-06-15 13:50:16

Is it possible to use an existing qcow2 image file (which is not created by image-builder), then pass that file as input to image-builder, and image-builder could generate a new image file, which would be cluster API compatible?

jsturtevant

2023-06-15 19:50:37

image builder uses ansible and packer to configure a vm. If there was a packer provisiner for qcow2 maybe? I don't know anything about the format

hislacker

2023-06-15 21:05:47

There's command to make qemu compatible image file via make_qemu command, I can't find a packer provisioner for qcow2.

Christophe Jauffret

2023-06-16 05:27:43

you just need to use an infrastructure provider who can consume qcow2 file as input, i use the Nutanix one like that, of course you need the corresponding infra.
we can also modify the qemu one to work like that i imagine

hislacker

2023-06-16 14:58:36

ok it would be great if there's some existing reference doc for this one..

Christophe Jauffret

2023-06-16 15:19:47

???

you have a Nutanix cluster ?

hislacker

2023-06-16 15:58:05

No. I am not asking if possible to generate an image to be useful on certain platform. I am asking if possible to take an existing qcow2 image, run imagebuilder process on that image to add relevant kubelet/kubeadm/certs in order for that existing qcow2 image to be k8s cluster api compatible. Per above nutanix provider doc, it seems it could generate an image usable for nutanix, but I could not find the place where it could add things to an existing image file.

Christophe Jauffret

2023-06-16 16:20:54

image-builder project generate CAPI ready image for a bunch of platform, there is no generic way
on which platform did you plan to run your image at the end ?

hislacker

2023-06-16 16:26:20

kubevirt

Christophe Jauffret

2023-06-16 18:17:09

ok i would say qemu is the closest target
but it build from an iso installer
what you can do is to modify
packer/qemu/packer.json
and replace all the iso stuff by using the disk_image parameter
look here

once done you can call your corresponding OS target

hislacker

2023-06-16 20:44:13

interesting! Thanks!

🙏:skin_tone_2: Christophe Jauffret

Yike Wang

2023-06-19 09:12:37

Hi, there are multiple case images . Where are they stored? How to understand amifilterowners , is it an account that can reach the base ami? 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 09:16:48

I'm not sure I understand what you're asking exactly.
The amifilterowners is used to filter the list of returned AMIs that match the amifiltername. This should result in just the base images we require. All the ones defined in the various .json files we have should be publicly available, if not then we need to fix those.
If you're looking to use your own base image then you can replace amifilterowners and amifiltername with valid filters that will result in your account finding the AMI you want to use.

Is that what you were looking for?

:_thank_you_: Yike Wang

Yike Wang

2023-06-19 09:24:09

I see. Who is responsible to maintain the base amis? And are the base amis avaliable in all the aws regions?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 09:34:59

That I'm not sure about unfortunately 😞
Hopefully they are "official" distro images but I'd need to check to be sure.

Yike Wang

2023-06-19 09:36:54

Thanks @Marcus Noble! Looking forward for your checkings 😀

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 09:37:40

Out of curiosity, what prompted the question? Are you looking to change something with CAPA?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 09:43:27

Amazon2 - Official

CentOS - Not sure but @swan updated the owner in this PR -

Flatcar - Official

RHEL8 - Official

RockyLinux - Official

Ubuntu - Official

Windows - Official

👍 richcase, Yike Wang, Paokrab

Yike Wang

2023-06-19 10:16:04

What does "Official" mean? Does it mean they are available across all aws regions or we don't need take care its mantainece or something?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 10:17:34

Official here means the account is run by either AWS themselves (AWS Marketplace) or is the account suggested to use by the distro's own documentation.
None of those images are without our control.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 10:21:07

**Within our control.

Yike Wang

2023-06-19 10:27:09

Got it, thanks @Marcus Noble!

Yike Wang

2023-06-19 10:29:00

Asking it is because our account that is responsible for making and publishing AMIs is getting migrated. So I think more on the base ami 🙂

👍 Marcus Noble

Slackbot

2023-06-19 15:30:25

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2023-06-19 15:32:50

I'll not be around for this one I'm afraid. I've not long got back from my holiday and I'm a bit all over the place atm.

I've not got anything major to raise for this one anyway.

WRT what @Marcus Noble has put in the agenda so far, I'll simply say I agree with all 😄

👍 mboersma, Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 18:12:47

@mboersma I took a little look at what we could do with the Makefile include. It's possible but we'd need to do a lot of updating to the Makefile tasks to make use of $(CURDIR) when referencing any files. I'm not confident enough to make all those changes as I don't understand a good chunk of that Makefile 😅
If you're interested, the basic idea would be...
In the root Makefile:

[...existing code...]

export CWD=images/capi
include $(CWD)/Makefile

In the images/capi/Makefile:

CWD ?= ./
CURDIR := $(realpath $(CWD))

(and then update all references to also use $(CURDIR))

mboersma

2023-06-20 16:41:39

@Marcus Noble that does sound a bit fiddly, but not complicated. Maybe we could try it out and see how it acts IRL.
(For context, Marcus and I were discussing flattening the repository, since everything important is nested down in /images/capi. We thought if we could somehow "proxy" the nested Makefile through the root one, it would relieve most of the pain. That is, typing make -C images/capi could be just make and we could avoid the disruption of actually moving those directory contents.)

👍 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-20 16:54:56

I’ll try and get a PR up at some point. The main thing that might cause problems is if we run any scripts that expect the cwd to be images/capi

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-23 08:11:08

@mboersma I finally for to putting the PR together. I'm not 100% sure I've caught everything but the handful of tasks I checked worked as expected.

GitHub

Allow running Make tasks from root directory by AverageMarcus · Pull Request #1211 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adds the ability to run all the Make tags from the root of the repo without the need to do make -c images/capi Which issue(s) this PR fixes (optional, in fixes #...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1211

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-23 08:11:59

As an aside... that Makefile is horrible to work with. There's SO MUCH. 😆

mboersma

2023-06-26 20:50:20

I know, it's huge and hard to navigate. I'll try out the make changes soon, sorry to let it dangle @Marcus Noble. (And thanks!)

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 21:20:32

It's ok. There's no rush for it. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 19:03:07

Hey y'all, is anyone still building Ubuntu-18.04 images with image-builder?

We're currently removing it from GCP because the base image is no longer available and wondering if we should remove it from all providers, even those that still have base images available. Ideally we'd like to clear out old OS versions but if there's still a need for it we'll keep it in image-builder for the time being. 🙂

➕ dims

:kubernetes_intensifies: dims

✂️ mboersma

Drew Hudson-Viles

2023-06-19 19:03:49

Not on my side. We're on 22.04 minimum now.

mboersma

2023-06-20 16:27:49

We stopped building 18.04 for CAPZ in April, when it went out of support.
The only reason I haven't made a PR to remove it from azure/ is this:

With an Ubuntu Pro subscription, your Ubuntu 18.04 LTS deployment can receive Expanded Security Maintenance (ESM) until 2028.

mboersma

2023-06-20 16:29:19

So theoretically, there could be an enterprise user with an Ubuntu Pro sub building their own images (as we strongly recommend, the "reference images" we build aren't updated with CVE fixes for one).

mboersma

2023-06-20 16:33:20

Although this theoretical user can and probably has forked the image-builder repo, so they can continue using the 18.04 make targets if we remove them.
I think for the vast majority of use cases, it's at best clutter and at worst a liability to keep all the 18.04-related stuff.
Evidently I've just talked myself into removing 18.04 globally, but I'm curious to hear other opinions still.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-20 16:57:02

I agree with your reasoning. It’s always possible to use an older version of image-builder if needed. Or adding a custom packer json.
If no one comes forward with a strong case for keeping it then let’s clear it out from all providers.

👍 mboersma, Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-19 20:11:03

Could anyone confirm for me if the qemu images can / are ever used on VMWare platforms?
I've opened a pull request to address this issue about open-vm-tools being installed in qemu images due to us sym-linking some files between providers but I want to make sure first that removing this isn't going to cause issues for anyone.

tormath1

2023-06-21 13:51:00

@tormath1 has joined the channel

tormath1

2023-06-21 14:04:21

Hello folks, I'm trying to build a Flatcar image using QEMU builder and it fails with this:

$ make OEM_ID=openstack build-qemu-flatcar
...
    qemu:
    qemu: PLAY [all] *
    qemu:
    qemu: TASK [Gathering Facts] 
    qemu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: command-line line 0: keyword identitiesonly extra arguments at end of line", "unreachable": true}
    qemu:
    qemu: PLAY RECAP *
    qemu: default                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    qemu:
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
==> qemu: Deleting output directory...
Build 'qemu' errored after 7 minutes 55 seconds: Error executing Ansible: Non-zero exit status: exit status 4

It was working fine previously, I suspect the OpenSSH upgrade of my system:

$ ansible --version
ansible [core 2.15.1]
  config file = /home/mathieu/github/kubernetes-sigs/image-builder/images/capi/ansible.cfg
  configured module search path = ['/home/mathieu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/mathieu/github/kubernetes-sigs/image-builder/images/capi/.env/lib/python3.11/site-packages/ansible
  ansible collection location = /home/mathieu/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/mathieu/github/kubernetes-sigs/image-builder/images/capi/.env/bin/ansible
  python version = 3.11.3 (main, Jun  2 2023, 13:54:39) [GCC 12.2.1 20230428] (/home/mathieu/github/kubernetes-sigs/image-builder/images/capi/.env/bin/python)
  jinja version = 3.1.2
  libyaml = True
$ ssh -V
OpenSSH_9.3p1, OpenSSL 1.1.1u  30 May 2023

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-21 14:24:10

Are you using the latest image-builder from the master branch or a tagged release?
Are you able to check if you experience the same issue using the Docker image?

tormath1

2023-06-21 14:47:25

I'm on master
Let me check with Docker but I need to rebuild an image to use root user instead of imagebuilder (see: )

tormath1

2023-06-21 15:26:46

That seems better from Docker - I see the image is using a different OpenSSH version:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022

Let me see if I can get the faulty command.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-21 15:28:15

Ok, so that at least confirms that the issue is related to OpenSSH being updated. Not sure how we'll tackle that thinking

tormath1

2023-06-21 16:01:30

Mm, maybe Ansible actually. I found this:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-21 16:03:50

Hmm... so maybe we just need to bump Ansible to v2.15.1?
Do you mind trying again with this value bumped up?

GitHub

image-builder/images/capi/hack/ensure-ansible.sh at 86a089475fe4dc47ae132b0b1e867c8d640089e0 · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/86a089475fe4dc47ae132b0b1e867c8d640089e0/images/capi/hack/ensure-ansible.sh#LL25C18-L25C18

tormath1

2023-06-21 16:07:56

I'd say the opposite actually: the version 2.15.1 does not currently work with image-builder.
Let me try with your version !

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-21 16:11:10

Oh sorry, I misunderstood.

tormath1

2023-06-21 16:22:23

That works fine with 2.11.5 so it's not OpenSSH related but Ansible related

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-21 16:24:32

Ok. That's good to know at least. Hopefully it gets resolved in Ansible before you do another upgrade of it in image-builder.

tormath1

2023-06-21 16:29:10

I'm trying to see if we could not send a similar patch: for SSH extra arguments ()

tormath1

2023-06-21 16:40:31

Anyway, thanks for your help 💪

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-21 16:43:47

No worries 🙂 Glad you managed to figure it out. 😄

Wolodja Wentland

2023-06-21 16:49:14

@Wolodja Wentland has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 12:56:28

📣 Announcement!
I am about to begin the master -> main branch rename as outline in this issue.
If all goes smoothly nothing should be effected as PRs and git references should update automatically. If you notice any problems please let me know in the thread. 🙂
I'll announce again once complete so people can update their own checkout out repos / forks if they choose.

:emoji_k8s_loft: Drew Hudson-Viles, Ricky Sadowski, mboersma, jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 12:57:13

For reference / info:

Screenshot 2023-06-22 at 12.57.01 pm.png

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 12:58:19

To update your local checkout out copy:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 13:05:55

😒 The test-infra PR deciding to re-run all the checks again after removing the hold label is not ideal.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 13:22:48

Held PRs now merged.
Just waiting on confirming that CI jobs still work as expected. (See )

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)

Is it possible for me (or someone else) to manually trigger a periodic test? I need to confirm things still work following a master->main branch rename. Specifically it'd be good if I could re-run <a href="https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-image-builder-gcp-all-nightly/1671806324221939712">this job</a> with the latest config yaml but Prow states I don't have permissions to do so. 😞

Original URL: https://kubernetes.slack.com/archives/C09QZ4DQB/p1687436517311499

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 13:38:58

Both PR and periodic jobs are able to run correctly! 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 13:40:11

Rename complete! ✅ 🎉
If you experience any issues you believe is related to the rename please don't hesitate to let us know. 🙂 If you'd like to update your local copy there's commands in this thread you can run.

:thank_you: mboersma, jsturtevant

:high5: Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 13:41:14

I've opened a PR in test-infra to allow image-builder maintainers to re-run our periodic jobs via Prow -

GitHub

Allow image-builder maintainers to re-run their periodics by AverageMarcus · Pull Request #29906 · kubernetes/test-infra

Enables those in the image-builder-maintainers GitHub group to be able to re-run the image-builder periodic job via the Prow UI.

Original URL: https://github.com/kubernetes/test-infra/pull/29906

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 13:56:06

@mboersma Did you get anywhere with the Netlify config update? The book is still working (as I expected) but I'm not sure about new changes being built and deployed. I've opened a cleanup PR that we can use to test the change when we're ready to do so.

GitHub

Clean up book intro page by AverageMarcus · Pull Request #1208 · kubernetes-sigs/image-builder

What this PR does / why we need it: Removes blank sections from book into page and updates the description to align more with what we also have on the readme. Which issue(s) this PR fixes (optional...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1208

mboersma

2023-06-22 17:34:18

mboersma

2023-06-22 17:49:32

I think it's all done. We can merge the cleanup PR to make sure.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 18:35:41

Already done 🙂 And confirmed it worked! 😄

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-22 18:36:11

Oh, just realised you sent this before I did the PR 😆 My notifications don't seem to be working right.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 08:26:21

Hey y'all, I'd like to get peoples thoughts on using rolling-version base images (within a set major version) and if people think this is appropriate or not for image-builder?
The question comes from this PR and originally this comment. The RockyLinux links we're currently using in Nunatix are dead but have been moved elsewhere. The PR is suggesting changing them to point to the latest of the major release which would mean we don't need to keep updating the value but it does mean that re-runs of the same image-builder configuration could result in different images being built. What does everyone think?

Christophe Jauffret

2023-06-26 08:29:17

Actually Image-builder inside his OS task is performing yum update / apt update during each build.
so each result is different with always the last security update.
Using the rolling-version as source will give the exact final result and even better will improve the build time because no need to download package twice.

so i approved :-D

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 08:32:27

Yeah, that is actually a really good point. I'm not sure we actually do this with other OSs (e.g. Ubuntu) but we definitely do with RHEL.

Christophe Jauffret

2023-06-26 08:33:29

same with ubuntu

- name: perform a dist-upgrade
  apt:
    force_apt_get: True
    update_cache: True
    upgrade: dist
  register: apt_lock_status
  until: apt_lock_status is not failed
  retries: 5
  delay: 10

Christophe Jauffret

2023-06-26 08:33:45

in images/capi/ansible/roles/setup/tasks/debian.yml

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 08:33:49

Yup! Just found that as you posted 🙈

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 08:33:59

Ok, I'm being convinced 😆

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 08:34:30

Lets leave it a little just in case anyone wants to weigh in but if not I'm happy to approve that PR 🙂

Christophe Jauffret

2023-06-26 08:35:40

security first, and also sustainable IT improvement 😎
…. and i don’t like wait

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 08:45:36

The other downside I see to using the lastest base image is we can't use checksum to ensure the image is as expected. But was we're not currently doing that for Nutanix anyway I guess it doesn't matter.

Christophe Jauffret

2023-06-26 08:46:46

yes agree

Christophe Jauffret

2023-06-26 08:47:47

was thinking to implement external checksum file support for this kind of case

mboersma

2023-06-26 16:26:43

I'm ok with using latest, because we already don't have 100% recreatable builds, since Ubuntu and others can already update underneath us (and I was planning to use "latest" for Mariner Linux support).

The tradeoff is obviously that a given CI run might break, but we already see that occasionally (usually from an old distro rusting too much, like Rocky Linux here).

👍 Marcus Noble, jsturtevant

👍:skin_tone_2: Christophe Jauffret

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 16:57:18

Works for me 🙂 I'm convinced. Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-06-26 16:58:20

lgtm approval

👍 mboersma

Ivo Perruci Neto

2023-06-26 18:06:21

@Ivo Perruci Neto has joined the channel

Ozhan Karaman

2023-06-26 20:39:45

@Ozhan Karaman has joined the channel

Slackbot

2023-07-03 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-03 16:35:09

Skipping the office hours this week due to low attendance and nothing pressing to discuss. 🙂

Jan

2023-07-05 19:47:56

@Jan has joined the channel

Krish Jain

2023-07-06 07:56:51

@Krish Jain has joined the channel

Patrick Enoux

2023-07-08 12:51:36

@Patrick Enoux has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-09 14:31:10

Could I please get a review on this small fix for Mac M1/2 users? 🙏
https://github.com/kubernetes-sigs/image-builder/pull/1215

GitHub

Add fallback for tools that don't have darwin/arm64 binaries by AverageMarcus · Pull Request #1215 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adds a helper function to fallback to amd64 on an M1/M2 Mac when a tool doesn't provide an arm64 binary. Which issue(s) this PR fixes (optional, in fixes #(, fix...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1215

:lgtm: mboersma

mboersma

2023-07-10 16:09:15

@Marcus Noble thanks for that!
Maybe we should do an image-builder release soon? Several good fixes, plus Mariner Linux and maybe updating to latest packer, seems like a good milestone to release.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-10 16:10:16

👍 Sounds good to me.
Anything else in the open PRs that'd be good to get in?

mboersma

2023-07-10 17:42:32

Nothing that jumps out at me. Maybe we wait for to merge and then do a release.

👍 Marcus Noble

mboersma

2023-07-12 21:15:42

I think we're at a good point to do a v0.1.17 image-builder release. I can kick that off this afternoon or tomorrow if we have consensus.

mboersma

2023-07-13 03:03:59

I tagged v0.1.17, here's the image promotion PR:

:thx_thanks: Marcus Noble

:approval: Marcus Noble

rahav jv

2023-07-12 14:15:04

Folks, I'm trying to set cdromtype to sata as a packer variable. I have set this is in a file which is passed to PACKERVAR_FILES . However when the packer vm comes up, the vm comes up with cdrom set as ide. Any suggestions on what i might be missing

kiran keshavamurthy

2023-07-12 19:03:32

Hey Rahav, cdromtype is not used in any of the builders so I guess it just defaults to ide . Hence passing the type has no effect. I suggest adding cdromtype var to the builders in packer-node.json , and pass the type from the from the os configs. For Photon-5 you might need sata I guess.

Debjit

2023-07-12 15:35:05

@Debjit has joined the channel

Debjit

2023-07-12 15:45:05

I could not understand the usage of image-builder Can't we use any ami with cluster-api ?

Drew Hudson-Viles

2023-07-12 15:49:00

image-builder is designed to set a bunch of sensible defaults as well as install the tools required for it to work with capi.
On top of that it supports multiple clouds, multiple distros and can be configured to install additional container images, nvidia drivers and more.

A base ami, or any other image from any other provider would require all of the bits the image builder project adds to be done manually.

It just takes the toil out of building images really.

Debjit

2023-07-12 15:51:24

Thanks @Drew Hudson-Viles. So we just cannot use any ami with cluster-api , right?

Drew Hudson-Viles

2023-07-12 15:51:56

Well you could, but you'd need to install things on top of it, like kubelets, containerd (or alternative) etc.

Drew Hudson-Viles

2023-07-12 15:52:19

capi isn't something you can just install and get all those things with it 🙂

Debjit

2023-07-12 16:58:40

@Drew Hudson-Viles Ok. Which means Cluster API does not install kubeadm and necessary applications and tools by itself. Like in the case of Kubespray.

Drew Hudson-Viles

2023-07-12 17:11:09

Yeah that's right. The best way to think about it is that Cluster API is just a collection of APIs that allow you to manage multiple clusters from a single management cluster - that could be a full blown one or kind that you run locally.

Debjit

2023-07-13 09:38:50

@Drew Hudson-Viles Got it. Thanks.

Debjit

2023-07-13 23:23:51

@Drew Hudson-Viles We can only use ami build for cluster-api Reading through the image-builder book and it seems it is not an easy route to build custom ami. As an example, i need to build aarch64 based ami as we are using AWS Graviton processor based instances, and there seems to be no way to build the required ami with image-builder at the moment. Can you advise some alternatives. Any documentation what needs to be provisioned for ami to be compatible with cluster-api , then I can probably create an ami manually for the time being.

Drew Hudson-Viles

2023-07-14 09:14:24

I can't say for certain as I've not used graviton to date and I'm out of AWS at the moment so can't do any testing.

You may be able to modify this to get it to work with graviton based AMIs but without any testing I cannot confirm for sure. Depending on the package manager it uses etc it may be required you use a different one.

If you do want to do it manually for now then there is a decent amount of documentation available. I'd recommend taking a look through the CAPI book and the cluster-api-provider repo

In the AWS CAPI provider readme there is information about using pre-baked AMIs which I'd suspect Amazon already provide to save any leg work on your side.

The final option I'd say is to read through the Ansible playbooks in the image builder repo and see what's being done at each stage and attempt to replicate that as part of the graviton AMI.

Debjit

2023-07-14 12:37:59

@Drew Hudson-Viles Your thoughts align. Thanks. The only issue is in image-builder the amd64/x86_64 is hardcoded. Let me do a little brainstorming and will let you know the results.

👍 Drew Hudson-Viles

mboersma

2023-07-13 15:37:17

Image-builder v0.1.17 is now available:
Thanks to all the contributors! thank_you

:parrotk8s: Marcus Noble, jsturtevant

:thx_thanks: Marcus Noble

🎉 Marcus Noble, Ozhan Karaman

Debjit

2023-07-14 12:39:22

How can I build arm64/aarch64 ami with image-builder?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-14 12:56:03

There's actually a PR I've just approved to add that support:

GitHub

Allow customizable kubernetes architecture by mpanduru · Pull Request #1222 · kubernetes-sigs/image-builder

What this PR does / why we need it: This change allows easy customization and override of the kubernetes_goarch variable when building an image with image-builder. With this change we can build im...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1222

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-14 12:57:25

Once that's merged in you should be able to by setting the kubernetes_goarch var. I think you also need to change the base image you use too. I'm not sure as I haven't done it myself.

Debjit

2023-07-14 23:39:39

👍

Debjit

2023-07-16 22:42:33

@Marcus Noble I cloned the repo but could not find kubernetes_goarch var

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-16 23:57:17

If you add kubernetes_goarch to your user provided packer vars then it’ll use the value you provide rather than the default. (See here for details on how if you’re not sure: https://image-builder.sigs.k8s.io/capi/capi.html#customization)

Debjit

2023-07-18 16:56:44

@Marcus Noble What about other hard coded values? There are a lot of places where am64/x86_64 is hard coded.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-18 17:33:01

Sorry, I didn’t realise there was still more outstanding work. Looks like we have this issue open to track it: https://github.com/kubernetes-sigs/image-builder/issues/936

GitHub

Support building arm64 CAPI images · Issue #936 · kubernetes-sigs/image-builder

Both Azure and AWS supports ARM64 VMs now, so I think it would be great if image-builder would be able producing arm64 compatible images. I see that there are already some parameters for selecting ...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/936

Tham Xun Hong

2023-07-14 15:48:25

@Tham Xun Hong has joined the channel

Debjit

2023-07-16 22:41:48

Can we have prebuilt image for Amazon Linux 2023 with Kubernetes v1.27.3. Tried building one with image-builder but epel package install failed, since AL 2023 does not allow the same.

Debjit

2023-07-18 17:15:43

@Marcus Noble Sure

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-17 07:59:03

Is anyone able to take a look at this PR and share your thoughts please? 🙂

GitHub

Remove VMWare open-vm-tools from qemu images by AverageMarcus · Pull Request #1201 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1201

👍 mboersma

Slackbot

2023-07-17 15:30:20

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2023-07-17 17:10:17

Thanks for attending everyone! If you couldn't attend, a recording of the office hours is available on YouTube here:

Debjit

2023-07-18 17:15:00

@Marcus Noble How can I make the images private only?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-18 17:30:03

What do you mean? What provider are you building for? What vars etc?

Debjit

2023-08-08 22:33:29

@Marcus Noble AWS. It is solved now.

Igar V

2023-07-19 15:59:54

@Igar V has joined the channel

Erkan Erol

2023-07-20 11:33:53

@Erkan Erol has joined the channel

Erkan Erol

2023-07-20 11:40:58

Hello. In the OVF file of Flatcar image for CAPV, variables for userconfig are missing. Is it intentional? While testing images via UI, they are very useful.

Here is the related place. I expect to see these variables there.

Erkan Erol

2023-07-20 11:43:04

Here is the related part in flatcarproductionvmware_ova.ovf



  
    
      Flatcar Container Linux Virtual Appliance
                      ovf:key="guestinfo.hostname" ovf:value="">
        Hostname
        Hostname
      
                      ovf:key="guestinfo.ignition.config.data" ovf:value="">
        Ignition/coreos-cloudinit data
        Inline Ignition config or coreos-cloudinit data (cloud-config or script)
      
                      ovf:key="guestinfo.ignition.config.data.encoding" ovf:value="">
        Ignition/coreos-cloudinit data encoding
        Encoding for Ignition config or coreos-cloudinit data (e.g., base64)
      
                      ovf:key="guestinfo.ignition.config.url" ovf:value="">
        Ignition/coreos-cloudinit config url
        URL to Ignition config or coreos-cloudinit data (cloud-config or script)
      
                      ovf:key="guestinfo.dns.server.0" ovf:value="">
        Primary DNS (only for coreos-cloudinit)
        Primary DNS (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.dns.server.1" ovf:value="">
        Secondary DNS (only for coreos-cloudinit)
        Secondary DNS (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.name" ovf:value="">
        Name for network interface 0 (only for coreos-cloudinit)
        Name for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.mac" ovf:value="">
        MAC for network interface 0 (only for coreos-cloudinit)
        MAC for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.dhcp" ovf:value="no">
        DHCP support for network interface 0 (only for coreos-cloudinit)
        DHCP support for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.role" ovf:value="public">
        Role for network interface 0 (only for coreos-cloudinit)
        Role for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.ip.0.address" ovf:value="">
        Main IP for network interface 0 (only for coreos-cloudinit)
        Main IP for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.ip.1.address" ovf:value="">
        Additional IP for network interface 0 (only for coreos-cloudinit)
        Additional IP for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.route.0.gateway" ovf:value="">
        Main route gateway for network interface 0 (only for coreos-cloudinit)
        Main route gateway for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.route.0.destination" ovf:value="">
        Main route destination for network interface 0 (only for coreos-cloudinit)
        Main route destination for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.route.1.gateway" ovf:value="">
        Additional route gateway for network interface 0 (only for coreos-cloudinit)
        Additional route gateway for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.0.route.1.destination" ovf:value="">
        Additional route destination for network interface 0 (only for coreos-cloudinit)
        Additional route destination for network interface 0 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.name" ovf:value="">
        Name for network interface 1 (only for coreos-cloudinit)
        Name for network interface 1 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.mac" ovf:value="">
        MAC for network interface 1 (only for coreos-cloudinit)
        MAC for network interface 1 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.dhcp" ovf:value="no">
        DHCP support for network interface 1 (only for coreos-cloudinit)
        DHCP support for network interface 1 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.role" ovf:value="private">
        Role for network interface 1 (only for coreos-cloudinit)
        Role for network interface 1 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.ip.0.address" ovf:value="">
        Main IP for network interface 1 (only for coreos-cloudinit)
        Main IP for network interface 1 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.route.0.gateway" ovf:value="">
        Main route gateway for network interface 1 (only for coreos-cloudinit)
        Main route gateway for network interface 1 (only for coreos-cloudinit)
      
                      ovf:key="guestinfo.interface.1.route.0.destination" ovf:value="">
        Main route destination for network interface 1 (only for coreos-cloudinit)
        Main route destination for network interface 1 (only for coreos-cloudinit)

nickperry

2023-07-21 00:26:03

@nickperry has joined the channel

nickperry

2023-07-21 00:28:35

In Kube 1.27, the in-tree kubelet credential provider for AWS was removed (). This followed GA of the external kubelet credential provider feature in 1.26.

At my organisation we pull most of our images from private ECR repos, so since this removal in 1.27, we need the external ecr-credential-provider binary in our CAPV OVAs.

What are the maintainers thoughts on including external credential provider binaries (such as ecr-credential-provider) in published images, such as the OVAs distributed at ?

nickperry

2023-07-21 08:40:06

Thanks. The ecr-credential-provider binary is 21MB.

nickperry

2023-07-21 08:41:08

I need to double check, but I don't think CAPA have included it yet either.

2023-07-21 10:29:20

@xd has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-07-24 06:54:24

I've opened an issue to track removing old end of life OSs from image-builder defaults:

GitHub

Remove EoL OSs from providers · Issue #1232 · kubernetes-sigs/image-builder

We've got a lot of OSs still included as defaults in our providers that have passed their end of life for support now. I think we should remove these from the project (people that rely on them can ...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1232

👍 mboersma

Dharmjit

2023-07-26 15:31:08

Hi Folks, I was trying build-node-ova-vsphere-photon-5 target to build ova with a different photon iso but the script is stuck at step vsphere-iso.vsphere: Waiting for SSH to become available... . When tried with the higher verbosity using FOREGROUND=1 PACKERLOG=1 , I get below in the output. I am running scripts in Ubuntu 20.04.5 LTS machine and using vSphere7 as the hypervisor.
Not sure how packer works but I could see the sshusername and ssh_password are already defined in images/capi/packaer/ova/packer-common.json. Any inputs?

==> vsphere-iso.vsphere: Waiting for SSH to become available...
2023/07/26 14:16:13 packer-builder-vsphere-iso plugin: [INFO] Attempting SSH connection to 10.xx.xx.xx:22...
2023/07/26 14:16:13 packer-builder-vsphere-iso plugin: [DEBUG] reconnecting to TCP connection for SSH
2023/07/26 14:16:13 packer-builder-vsphere-iso plugin: [DEBUG] handshaking with SSH
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: Keyboard interactive challenge:
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: -- User:
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: -- Instructions:
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: -- Question 1: Password:
2023/07/26 14:16:19 packer-builder-vsphere-iso plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password keyboard-interactive], no supported methods remain
2023/07/26 14:16:19 packer-builder-vsphere-iso plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.
2023/07/26 14:16:26 packer-builder-vsphere-iso plugin: [INFO] Attempting SSH connection to 10.xx.xx.xx:22...
2023/07/26 14:16:26 packer-builder-vsphere-iso plugin: [DEBUG] reconnecting to TCP connection for SSH
2023/07/26 14:16:26 packer-builder-vsphere-iso plugin: [DEBUG] handshaking with SSH

Dharmjit

2023-07-26 17:32:35

@kiran keshavamurthy tagging you if you can provide some inputs.

shravanr

2023-07-26 16:30:15

@shravanr has joined the channel

Slackbot

2023-07-31 15:30:17

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

Max Fedotov

2023-08-01 10:03:17

@Max Fedotov has joined the channel

opatrick

2023-08-07 11:54:45

In some Docker Images i found an spdx file, what is this file for?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-08-07 12:51:55

What file specifically are you referring to? I don’t know of any spdx files but you can find the Dockerfile here: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/Dockerfile it might give you some hints.

Jeremi Piotrowski

2023-08-07 14:08:54

spdx is a format for SBOMs (software bill of materials) so it would be metadata about contents and licenses of package inside a container image

❤️ opatrick

opatrick

2023-08-07 14:14:05

Ahh cool, is this a docker feature?

Jeremi Piotrowski

2023-08-07 15:49:31

im not sure what the source of the sbom is that you're seeing

Jeremi Piotrowski

2023-08-07 15:49:36

docker has some form of integration

Jeremi Piotrowski

2023-08-07 15:49:41

Docker Documentation

Generate the SBOM for Docker images

Viewing the Software Bill of Materials (SBOM) for Docker images

Original URL: https://docs.docker.com/engine/sbom/

❤️ opatrick

Abhay Krishna Arunachalam

2023-08-10 23:41:04

Does this raise any concerns for the usage of Packer in image-builder?
https://discuss.hashicorp.com/t/hashicorp-projects-changing-license-to-business-source-license-v1-1/57106
https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license

HashiCorp Discuss

HashiCorp projects changing license to Business Source License v1.1

Today, HashiCorp has announced a transition in license from the Mozilla Public License v2.0 (MPL 2.0) to the Business Source License v1.1 (BSL or BUSL) for future releases of all products and several libraries. HashiCorp APIs, SDKs, and almost all other libraries will remain MPL 2.0 HashiCorp is committed to continuing to develop our popular community products in the open with broad access and permissions for re-use. You can find answers to the most common questions in the FAQ published on the ...

Original URL: https://discuss.hashicorp.com/t/hashicorp-projects-changing-license-to-business-source-license-v1-1/57106

HashiCorp

HashiCorp adopts Business Source License

HashiCorp adopts the Business Source License to ensure continued investment in its community and to continue providing open, freely available products.

Original URL: https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license

Marcus Noble (k8s@marcusnoble.co.uk)

2023-08-11 08:30:40

Somewhat, yes.
I don’t understand enough to know if it will cause problems for image-builder but I suspect it could do.
I’ve added it as an agenda item to the next office hours meeting on Monday.

👍 jsturtevant, mboersma

jsturtevant

2023-08-11 16:45:30

we may need to get some help from the CNCF folks

mboersma

2023-08-11 17:09:50

Related: Bump Packer to v1.9.2

👍 Marcus Noble

:thx_thanks: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-08-11 17:20:46

https://kubernetes.slack.com/archives/C5P3FE08M/p1691750465350369?thread_ts=1691699636.105219&channel=C5P3FE08M&message_ts=1691750465.350369

Davanum Srinivas (https://kubernetes.slack.com/team/U0Y7A2MME)

@Marcus Noble for now, please pin the version of packer you use and open an issue to track what else we need to do about it.

Original URL: https://kubernetes.slack.com/archives/C5P3FE08M/p1691750465350369?thread_ts=1691699636.105219&channel=C5P3FE08M&message_ts=1691750465.350369

👍 jsturtevant

Marcus Noble (k8s@marcusnoble.co.uk)

2023-08-11 17:21:26

I’m AFK until Monday. If someone else has the time before then would you mind opening an issue to track this?

mboersma

2023-08-11 17:23:41

Sure, I can open an issue. Thanks Marcus!

:thx_thanks: Marcus Noble, jsturtevant

Abhay Krishna Arunachalam

2023-08-11 23:17:23

Thank you @Marcus Noble and @mboersma! I shall keep an eye on the issue

Marcus Noble (k8s@marcusnoble.co.uk)

2023-08-15 15:20:06

CNCF have published some guidance:

Doesn't really help us though. 😞 The two recommendations are:

Switch to an alternative - Is there one for Packer?

Freeze the component version - We've done this but it's a short-term solution at best

Marcus Noble (k8s@marcusnoble.co.uk)

2023-08-15 15:24:38

Image Builder Releases

The current release of Image Builder is [v0.1.19][] (September 13, 2023). The corresponding container image is `<a href='http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.19'>registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.19</a>`.

## Release Process

For more detail about image-builder project releases, see the [Image Builder Book][].

[v0.1.19]:
[Image Builder Book]:

GitHub

Release v0.1.19 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.19">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.19</a> What's Changed CAPC: Remove ConfigDrive from cloud-init data...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.19

feitnomore

2023-10-03 21:33:06

I am trying to add Ubuntu 23.04 to qemu, but I am failing miserably

fad3t

2023-10-04 07:03:52

maybe you can share some more details, it's hard to help just knowing it fails 😛

feitnomore

2023-10-04 11:20:09

I could achieve it

feitnomore

2023-10-04 11:20:20

it was a problem with apt, that was trying to use a local mirror

feitnomore

2023-10-04 11:20:28

I need to test the images generated though

feitnomore

2023-10-04 11:23:02

apt:
mirror-selection:
primary:
- uri:

feitnomore

2023-10-04 11:23:15

I've added this to my autoinstall cloud-config

feitnomore

2023-10-03 21:33:09

can anyone assist?

ARYAN

2023-10-04 15:55:44

@ARYAN has joined the channel

Alessandro Giorgio Togna

2023-10-04 18:40:32

I am trying to rebuild a CAPI image for k8s 1.26.6 using this values:
{
"kubernetessemver": "v1.26.6",
"kubernetesdebversion": "1.26.6-00",
}
however the make is failing since it cannot find kubectl/kubeadm 1.26.6-00.
Where do I get the list of available values for kubernetesdeb_version ?

Drew Hudson-Viles

2023-10-04 18:46:05

Try apt-cache madison kubeadm on a system where the repo available. This will yield the versions available.

Alessandro Giorgio Togna

2023-10-04 18:48:43

if I do that the 1.26.6-00 version is available, but make fails with a "cannot find the 1.26.6-00" version of the package

Alessandro Giorgio Togna

2023-10-04 18:49:44

for instance, the default version now is:
"kubernetesdebversion": "1.26.7-1.1"
which I cannot find with apt-cache

Drew Hudson-Viles

2023-10-04 18:50:37

Are you using the new apt repos on the machine on which you're running apt-cache?

Alessandro Giorgio Togna

2023-10-04 18:52:03

maybe not, I'll check now

Drew Hudson-Viles

2023-10-04 18:52:41

Cool 🙂 I suspect it may be the change in repos which is why you're not seeing it. Let me know if that doesn't work.

🙌 Alessandro Giorgio Togna

Alessandro Giorgio Togna

2023-10-04 18:56:06

thank you, I had the "wrong" repo where all the versions are -00

Alessandro Giorgio Togna

2023-10-04 18:56:32

I configured the new repo (and all versions are -1.1)

Drew Hudson-Viles

2023-10-04 18:56:40

Glad that's worked 🙂. It was a fairly recent change so could be easily missed.

Animesh Pandey

2023-10-05 05:33:27

@Animesh Pandey has joined the channel

aniruddha

2023-10-05 19:27:15

@aniruddha has left the channel

feitnomore

2023-10-06 20:13:18

I could build the Ubuntu 23.04 image, however, looks like cloud-init is not working well... no dhcp, no user created, no kubeadm

feitnomore

2023-10-06 20:13:42

I could log into the machine by creating a user by hand, if I run dhclient it gets ip

feitnomore

2023-10-06 20:13:55

hostname is not changed as well, it is localhost.localdomain

feitnomore

2023-10-06 20:14:18

I could mount the cloud-init disk /dev/vda, and I see it has a openstack dir with everything inside it

feitnomore

2023-10-07 00:23:41

(I am trying to build a qemu one)

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-09 13:04:36

We've got the image-builder office hours later today but currently the agenda is empty. I'm not aware of any topics that need discussing and if there's nothing added by 1 hour before the call I plan to cancel it and meet up in a couple weeks for the next sync instead.

✅ Drew Hudson-Viles

Slackbot

2023-10-09 15:30:14

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-09 15:37:26

Agenda empty. Cancelling office hours 🙂 See y'all in 2 weeks!

👍 mboersma, jsturtevant

mboersma

2023-10-11 17:21:34

Image-builder CI is currently broken due to a python update in the e2e environment. I think this PR fixes it:

:approval: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-12 08:28:38

I asked this on the PR but prob best to chat here...

Do you happen to know if the E2E environment changes were announced anywhere? Is there something we should be keeping an eye on for potential problems?

Arnaud (he/him)

2023-10-12 04:59:25

@Arnaud (he/him) has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-13 16:40:00

We've not done a release for the past month - anyone against doing a new one? Any changes people would like to get in before we do?
Changes pending release:

Drew Hudson-Viles

2023-10-13 16:40:57

Seems like a good idea to me

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-13 16:41:56

It's end of day for me and I'm not in a rush to have it done right now so I might give it a little time (e.g. until Monday) for people to speak up with any changes they want to get in.

Drew Hudson-Viles

2023-10-13 16:42:23

I don'yeah makes sense. No one wants a Friday release 😁

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-13 16:43:11

Right?! I have a habit of breaking clusters on Friday 😅

😆 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-16 07:58:47

I'm going to start the process of getting v0.1.20 released

🙌 Abhay Krishna Arunachalam

:ty: mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-16 08:23:34

Promotion PR -

GitHub

Image promotion for scl-image-builder v0.1.20 by AverageMarcus · Pull Request #5978 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.20 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers "@AverageMarcus @jsturtevant @kkes...

Original URL: https://github.com/kubernetes/k8s.io/pull/5978

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-16 08:27:13

And docs updates:

🙌 Drew Hudson-Viles

Siddhiprada Mohapatro

2023-10-14 09:05:31

@Siddhiprada Mohapatro has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-16 13:10:55

Image-builder v0.1.20 is now available:
Thanks to all contributors!

GitHub

Release v0.1.20 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.20">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.20</a> What's Changed Update docs for v0.1.19 release by @mboersma ...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.20

🎉 mboersma, RAKESH BOINAPALLY, Anurag

Amim Knabben

2023-10-16 18:24:47

hey folks anyone else having issues with ansible-galaxy?

Amim Knabben

2023-10-16 18:26:24

  41 ~ │ ansible-galaxy -vvv collection install <br>  42   │   community.general <br>  43   │   ansible.posix <br>  44   │   'ansible.windows:>=1.7.0' <br>  45   │   community.windows

community.general is failing for me when running hack/ensure-ansible.sh
ERROR! Unexpected Exception, this is probably a bug: '/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'

with the 2.11.5 the workaround is to add on ansible.cfg

Amim Knabben

2023-10-16 18:29:34

GitHub

Unable to install some galaxy collections · Issue #14495 · ansible/awx

Please confirm the following I agree to follow this project's code of conduct. I have checked the current issues for duplicates. I understand that AWX is open source software provided for free and ...

Original URL: https://github.com/ansible/awx/issues/14495#issuecomment-1752708302

Amim Knabben

2023-10-16 18:29:50

should ansible be bumped instead?

Amim Knabben

2023-10-16 19:07:16

oh i'm using an old version, seems the latest is 2.15 already

mboersma

2023-10-16 19:07:41

Oops sorry, I meant to reply in this thread.

The problem I ran into is this, not sure if that's exactly the same as yours:

But image-builder does effectively require ansible 2.15.5 now, although the script doesn't check versions, just that ansible exists.

👍 Amim Knabben

mboersma

2023-10-16 19:08:35

Maybe we should open an issue to change the ensure-ansible.sh script to also check the version.

Amim Knabben

2023-10-16 19:09:33

yea maybe a warning

Amim Knabben

2023-10-16 19:09:53

the Unexpected Exception without -vvv is not useful at all

Amim Knabben

2023-10-16 19:09:56

let me open it

Amim Knabben

2023-10-16 19:17:23

GitHub

Make ensure-ansible.sh check Ansible version · Issue #1323 · kubernetes-sigs/image-builder

Is your feature request related to a problem? Please describe. Ansible changed their collections server, it's needed Ansible >=2.13.9 to use it. Otherwise a not so clear error is raised when previo...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1323

mboersma

2023-10-16 19:20:43

Thank you!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 08:35:30

I don't think we want to use the old-galaxy do we? The solution we want for image-builder is version checking with a user-friendly error if found to be less than 2.13.9 right?

Amim Knabben

2023-10-19 11:00:36

@Marcus Noble the problem is breaking all the

Amim Knabben

2023-10-19 11:00:49

but forward I think it makes sense to fail close

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 11:02:33

Oh sorry, this is an external change that is now causing issues?

Amim Knabben

2023-10-19 13:14:41

yes, its an ansible server

Travis Holton

2023-10-17 02:41:26

@Travis Holton has joined the channel

feitnomore

2023-10-17 13:50:44

I
am using Cluster-API with Kubevirt and Kubeadm... question: Is there
any documentation on what exactly the image needs to look like to work
on Cluster API? I've built a few images with image-builder that are not
working and I am not sure how to troubleshoot.... I am not sure if there is a package missing, a service that was not
started, or a config file... Is there any document about the
requirements on the image?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 08:36:56

Hey Marcelo, it might be useful to explain what error you're seeing and what vars / make target you're using, etc.
I don't have any experience with Kubevirt so might not be able to help you myself but might be able to point you in the right direction.

feitnomore

2023-10-19 10:53:41

Any tips are pretty much appreciated

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 10:56:36

I can't help without the information I asked for.

What error are you seeing?

What make target are you running?

What variables are you providing (if any)?

Also, would be good to know if you're using the latest released version and if you're running it directly or with the container image.

feitnomore

2023-10-19 11:02:12

ok, haven't seem those bullet points, sorry

feitnomore

2023-10-19 11:03:04

let's go with the easier one... I've copied ubuntu 22.04 to 23.04, and I am running qemu-ubuntu for it

feitnomore

2023-10-19 11:03:17

the image builds ok

feitnomore

2023-10-19 11:03:32

but kubeadm doesn't bootstrap on it, the control plane doesn't get up

feitnomore

2023-10-19 11:03:38

I am not sure where to look for errors

feitnomore

2023-10-19 11:04:23

export PACKER_FLAGS="--var 'kubernetes_rpm_version=1.27.6-0' --var 'kubernetes_semver=v1.27.6' --var 'kubernetes_series=v1.27'  --var 'kubernetes_deb_version=1.27.6-00' --var 'disk-size=6144'"

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 11:04:39

Oh right, so you're trying to build a new OS that we don't yet support?

feitnomore

2023-10-19 11:05:07

I think that is the easy one... I've used the same flags, to build-qemu-flatcar

feitnomore

2023-10-19 11:05:18

the image boots, however, I never get the cluster up

feitnomore

2023-10-19 11:05:33

I can troubleshoot both... those are the ones that I've been trying so far

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 11:07:55

Gotcha!
Ok, so hard to say what could be going on. If you haven't done so already I recommend taking a look at the boot log to see if there are any failures that stand out in there (e.g. failure to pull something from the internet or start a service). If that all looks ok I'd suggest taking a look at journalctl -f and look out for services (specifically kubeadm related) that are failing to start.

Did the build log output have any warning that might be related?

feitnomore

2023-10-19 11:16:50

I haven't seen

feitnomore

2023-10-19 11:22:45

my best clue, when it comes to ubuntu 23.04 is that it might have something to do with the fact that the NIC name changes... its not in enps, its like ens**

feitnomore

2023-10-19 11:23:08

and I see there is a netplan file the image-build uploads over there, that I think might be related to the problem

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 12:07:26

Oh yeah that could possibly be the problem. Have you tried updating this file? https://github.com/kubernetes-sigs/image-builder/blob/c70b22baba58b89cf5d6561bc8e2[…]nsible/roles/sysprep/files/etc/netplan/51-kubevirt-netplan.yaml

feitnomore

2023-10-19 14:50:18

generating a new image now

feitnomore

2023-10-19 15:24:39

Pastebin

root@dev-box:/local/src/github.com/image-builder-kubevirtmanager/images/capi# ex - Pastebin.com

<a href="http://Pastebin.com">Pastebin.com</a> is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Original URL: https://pastebin.com/QzFWuqYD

feitnomore

2023-10-19 15:24:44

now I'll try it

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 15:52:49

Did that run successfully then?

feitnomore

2023-10-19 15:54:11

looks like 🙂

feitnomore

2023-10-19 15:54:30

my system is a little bit complex, but I'm importing the image right now into my KubeVirt Cluster to test

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 15:56:14

🤞

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 15:56:35

🤔 Need to have a think about how best to handle this for different OS's

feitnomore

2023-10-19 18:32:24

I think after some troubleshooting that I could get it working

feitnomore

2023-10-19 18:32:51

control plane is up, waiting for worker nodes

feitnomore

2023-10-19 18:36:15

gotta figure out what went wrong in the build process, to have it fixed

feitnomore

2023-10-19 18:36:40

but I think the trick was done by running sudo cloud-init clean --machine-id

feitnomore

2023-10-19 18:36:53

and systemctl enable cloud-init

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-19 18:48:02

Glad to hear you're making progress! 😄

feitnomore

2023-10-19 18:50:08

I might have done a lot of mess over here, but its working

😆 Marcus Noble

feitnomore

2023-10-19 18:50:10

lol

feitnomore

2023-10-19 19:34:45

so, for now, I have a guess it has something to do with the cloud-init clean, and not the netplan file

feitnomore

2023-10-19 19:35:04

cause all my other images work, and they don't have the netplan file, as I'm building as a plain simple qemu image

feitnomore

2023-10-19 19:35:15

and I am not setting kubevirt=true

feitnomore

2023-10-17 13:50:59

Example: flatcar isn't working

Alex B.

2023-10-19 13:38:14

@Alex B. has joined the channel

discostu

2023-10-20 07:23:54

@discostu has joined the channel

Slackbot

2023-10-23 15:30:19

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-23 15:33:06

☝️ Any topics for today?

mboersma

2023-10-23 16:02:32

Nothing in particular from me.

mboersma

2023-10-23 16:02:54

But I'm happy to meet if anyone has a discussion topic. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-23 16:15:22

👍 I'm about so can join and see if there's anything to discuss 🙂

👍 mboersma

Shalin Patel

2023-10-23 17:33:37

@Shalin Patel has joined the channel

Yike Wang

2023-10-25 14:50:18

when building 1.28.2 image based on v0.1.20 release, I meet:

    amazon-ebs.{{user build&#95;name}}: TASK [include_role : kubernetes] *
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Add the Kubernetes repo key] *
    amazon-ebs.{{user build&#95;name}}: changed: [default]
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Add the Kubernetes repo] *
    amazon-ebs.{{user build&#95;name}}: changed: [default]
    amazon-ebs.{{user build&#95;name}}:
    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Install Kubernetes] *
    amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "No package matching 'kubelet' is available"}

Any idea? Should I change some registry source or something, or downgrade something? Thanks!

Yike Wang

2023-10-25 14:50:58

my config file:

 /Users/yikew/Working/capa/image-builder/1.28.2/config.json
{
 "kubernetes_series": "1.28",
 "kubernetes_semver": "v1.28.2",
 "kubernetes_rpm_version": "1.28.2-0",
 "kubernetes_deb_version": "1.28.2-00",
 "kubernetes_source_type": "pkg",
 "kubernetes_http_source": "",
 "kubernetes_rpm_repo": "",
 "kubernetes_rpm_gpg_key": "\" \"",
 "kubernetes_rpm_gpg_check": "True",
 "kubernetes_deb_repo": "\" kubernetes-xenial\"",
 "kubernetes_deb_gpg_key": "",
 "kubernetes_container_registry": "registry.k8s.io",
 "kubernetes_load_additional_imgs": "false",
 "kubeadm_template": "etc/kubeadm.yml",
 "containerd_version": "1.7.6",
 "containerd_sha256": "20da1f2252d2033594b06e1eb68dd4906ff439f83f1003b7ebacdffcb4b95bdc"
}

Yike Wang

2023-10-25 15:10:11

it happens when building build-ami-ubuntu-2004 and build-ami-ubuntu-2204
build-ami-centos-7 and build-ami-amazon-2 work fine.

Yike Wang

2023-10-25 16:02:19

downgrade image-builder to v0.1.17 and then it works on build-ami-ubuntu-2004 and build-ami-ubuntu-2204 finally.

mboersma

2023-10-25 17:36:39

@Yike Wang I think this is because the format of kubernetesdebversion and kubernetesrpmversion changed slightly in .

The packages now come from the approved pkgs.k8s.io repository, but Kubernetes is using new tooling to publish. So I think this will work with image-builder v0.1.20:

"kubernetes_rpm_version": "1.28.2",
"kubernetes_deb_version": "1.28.2-1.1",

👍 Drew Hudson-Viles

Yike Wang

2023-10-28 15:56:01

I follow the new configs in . , but I'll hit:

    amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "Failed to download key at : HTTP Error 403: Forbidden"}

my config:

{
 "kubernetes_series": "1.28",
 "kubernetes_semver": "v1.28.3",
 "kubernetes_rpm_version": "1.28.3",
 "kubernetes_deb_version": "1.28.3-1.1",
 "kubernetes_source_type": "pkg",
 "kubernetes_http_source": "",
 "kubernetes_rpm_repo": " user kubernetes&#95;series }}/rpm/",
 "kubernetes_rpm_gpg_key": " user kubernetes&#95;series }}/rpm/repodata/repomd.xml.key",
 "kubernetes_rpm_gpg_check": "True",
 "kubernetes_deb_repo": " user kubernetes&#95;series }}/deb/",
 "kubernetes_deb_gpg_key": " user kubernetes&#95;series }}/deb/Release.key",
 "kubernetes_container_registry": "registry.k8s.io",
 "kubernetes_load_additional_imgs": "false",
 "kubeadm_template": "etc/kubeadm.yml",
 "containerd_version": "1.7.6",
 "containerd_sha256": "20da1f2252d2033594b06e1eb68dd4906ff439f83f1003b7ebacdffcb4b95bdc"
}

do you have idea

Abhay Krishna Arunachalam

2023-10-28 21:52:49

Try changing the kubernetes_series parameter to v1.28, with the leading v. That should fix it.

curl -ILs -o /dev/null -w "%{http_code}" https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.28/deb/Release.key
403

curl -ILs -o /dev/null -w "%{http_code}" https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/Release.key
200

Yike Wang

2023-10-29 02:37:27

I always use 1.28 as kubernetesseries without problem, never notice it. Thank you! @Abhay Krishna Arunachalam it works!

But for amazon image noproxy=* make build-ami-amazon-2 , there is no satisfied cri-tools found in the new repositories

amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Install Kubernetes] *
    amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "changes": {"installed": ["kubelet-1.28.3", "kubeadm-1.28.3", "kubectl-1.28.3", "kubernetes-cni-1.2.0"]}, "msg": "Error: Package: kubeadm-1.28.3-150500.1.1.x86_64 (kubernetes)\n           Requires: cri-tools >= 1.28.0\n           Available: cri-tools-1.25.0-1.amzn2.0.1.x86_64 (amzn2-core)\n               cri-tools = 1.25.0-1.amzn2.0.1\n           Available: cri-tools-1.26.1-1.amzn2.0.1.x86_64 (amzn2-core)\n               cri-tools = 1.26.1-1.amzn2.0.1\n           Available: cri-tools-1.26.1-1.amzn2.0.2.x86_64 (amzn2-core)\n               cri-tools = 1.26.1-1.amzn2.0.2\n", "rc": 1, "results": ["Loaded plugins: extras_suggestions, langpacks, priorities, update-motd\n227 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package kubeadm.x86_64 0:1.28.3-150500.1.1 will be installed\n--> Processing Dependency: cri-tools >= 1.28.0 for package: kubeadm-1.28.3-150500.1.1.x86_64\n---> Package kubectl.x86_64 0:1.28.3-150500.1.1 will be installed\n---> Package kubelet.x86_64 0:1.28.3-150500.1.1 will be installed\n---> Package kubernetes-cni.x86_64 0:1.2.0-150500.2.1 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}

Should I file an issue somewhere?

feitnomore

2023-10-25 18:56:27

I've got PR opened

GitHub

QEMU RockyLinux 9 and Ubuntu 23.04 by feitnomore · Pull Request #1332 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1332

feitnomore

2023-10-25 18:56:41

if anyone wants to talk about it, feel free to ping me

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-27 08:10:26

Is there anyone with an understanding / experience with cloud-init in Ubuntu that might be able to help out @Shalin Patel with this issues - ? It looks like something changed in the 23.3.1-0ubuntu1~20.04.1 release of cloud-init that broke builds for cluster-api-provider-aws. 😞

GitHub

Cloud-init fails for ubuntu 20.04 base AMI and Cloud-init version '23.3.1-0ubuntu1~20.04.1' · Issue #1333 · kubernetes-sigs/image-builder

What steps did you take and what happened: The latest cloud-init version 23.3.1-0ubuntu1~20.04.1 that is shipped with base AMI for Ubuntu 20.04 is unable to run boothook <https://cloudinit.readthedo>...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1333

Drew Hudson-Viles

2023-10-27 09:17:08

I'm not familiar enough with it to be able to say what's changed to be honest. If I get chance this weekend I'll have a look but it does look like something in the package has changed to cause this since they can downgrade and get it working.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-27 09:17:54

Yeah exactly. But I've also no clue about that package so was hoping to get some insight from the hive mind of Slack 😄

Shalin Patel

2023-10-27 17:09:57

Thank you for bringing it to attention. This issue was reported and observed by us in CAPA. CAPA members are looking into it.

Robert Van Voorhees (https://kubernetes.slack.com/team/U2B8TBG58)

We're having issues with cloud init 23.3.1, things were working with 23.2.2, has anybody experienced similar issues? Seems like it's not running properly, this is on capa ec2

Original URL: https://kubernetes.slack.com/archives/C8TSNPY4T/p1697637405978589

Erkan Erol

2023-10-31 12:07:44

I built an OVA image (flatcar-stable-3602.2.1-kube-v1.24.12) and tried to import it to vSphere. I hit this issue

Issues detected with selected template. Details: - 51:7:VALUE_ILLEGAL: Value ''VirtualSCSI'' of ResourceSubType element not found in [lsilogic, lsilogicsas]. - 94:7:VALUE_ILLEGAL: Value ''3'' of Parent element does not refer to a ref of type DiskControllerReference.

When I compare the OVF file in my image(flatcar-stable-3602.2.1-kube-v1.24.12) and the current upstream image (flatcar-stable-3374.2.5-kube-v1.24.11), there is no diff related to VirtualSCSI

However, I found this diff.

# upstream image
    

# my image

I manually change this and tried again. It worked. Then I check the repo and found this change https://github.com/kubernetes-sigs/image-builder/commit/d410869b84532819f5299a77188c4af20b5d283d#diff-e3ec89888eeec052751[…]e01561d14c5ffd22fb677a2d

I tested upstream 1.28.0 that has the change. It failed too. I manually changed the osType and it worked again. Is there anyone who is aware of this issue?

GitHub

Correction on OS_TYPE · kubernetes-sigs/image-builder@d410869

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/commit/d410869b84532819f5299a77188c4af20b5d283d#diff-e3ec89888eeec052751215d2aa1a88c0b402bdb2e01561d14c5ffd22fb677a2d

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-31 15:24:19

This is the PR that introduced the change:

@Yiyi Zhou as the author of that PR are you able to offer any insight?

@Erkan Erol Can you confirm what version of vSphere you're working with? If I'm understanding this page correctly it looks like otherLinux64Guest was introduced in v5.0. But I'm just trying to search for related things and don't have much actual insight into vSphere stuff as you know 😅

GitHub

Modify ova/rockylinux OS type by zyiyi11 · Pull Request #1103 · kubernetes-sigs/image-builder

What this PR does / why we need it: According to newly updated GuestOsIdentifier, rockylinux_64Guest is added on Jan/12/2022. Modify images/capi/packer/ova/rockylinux-8.json to utilize rockylinux_...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1103

Yiyi Zhou

2023-10-31 17:45:35

Yeah otherLinux64Guest is introduced since 5.0. If this change is breaking backward compatibility, I will file to revert.

Erkan Erol

2023-11-02 08:44:45

The version of vSphere I use is 7.0.3 and it gives the weird error I mentioned above for the images that contains otherLinux64Guest. Interesting. @Yiyi Zhou Have you ever tried to upload to vSphere an image after this change? What could I be missing?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 08:46:51

Weird 😕 I guess if the linux-64 one works I think you can set vsphereguestos_type in your provided vars to override it. I haven't actually checked through the code but that PR seems to suggest that would work. 🙂

Erkan Erol

2023-11-02 08:51:23

Really? Let me check.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 08:52:00

I think so. This line seems to suggest so:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 08:52:31

Actually, let me double check to be sure. 1 min.

Erkan Erol

2023-11-02 08:53:50

I didn’t see anything in the python script that respects this variable.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 08:53:53

ok, things get stranger. It looks like flatcar should be other3xLinux64Guest based on this:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 08:54:34

Which python script? It looks to be passed to Packer as guestostype in packer-node.json

Erkan Erol

2023-11-02 08:56:28

images/capi/hack/image-build-ova.py overrides OVF file based on this template

Erkan Erol

2023-11-02 08:57:06

It has a hardcoded dictionary here

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 08:58:25

Yeah just looking at that and I'm now even more confused 😕 Do you know where that builddata comes from? For Flatcar I would have expected it to be other3xLinux64Guest based on the link I posted above but there's no key in the map for that so I would expect it to end up with an empty string for the OSTYPE

Erkan Erol

2023-11-02 09:00:27

It is the custom_data in the packer-manifest.json

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 09:00:41

Oh, I'm starting to understand I think.
In this file:
guestostype is set in multiple places and using two different values, either user guest_os_type or user vsphere_guest_os_type
I don't know if that is correct.

Erkan Erol

2023-11-02 09:01:56

Since OSidmap doesn’t have any key-value pair to pick linux-64 , there is no way to set it right now. There should be another parameter to switch between old and new formats here

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-02 09:09:17

Yeah not sure how to handle that. Any thoughts?

Erkan Erol

2023-11-02 09:29:50

I am trying to understand why the image doesn’t work for me when my vSphere is 7.0.3. Maybe I am missing something.

Erkan Erol

2023-11-02 09:30:31

To support both cases, we need to extend images/capi/hack/image-build-ova.py a little bit. @Yiyi Zhou What do you think?

Erkan Erol

2023-11-06 17:12:08

Hi again. I checked this issue in detail again. As far as I understand, the problem is flatcar specific. Flatcar should be mapped to other3xLinux64Guest instead of otherLinux64Guest

I tested with vSphere 7.0.3.01400 and it worked.

In the official Flatcar image, the value is other3xLinux64Guest See

I downloaded the latest stable OVA image of Flatcar and checked it too. It is other3xLinux64Guest there too.

The value in flatcar.json is already other3xLinux64Guest See

The weird part is the relationship between guestostype , vsphereguestostype , image-build-ova.py . It is really confusing. I didn’t get how it works.

image-builder-ova.py respects guestostype but there are some re-assignments in builder configuration like
"guest_os_type": "{{user vsphere_guest_os_type}}",

for vsphere and vsphere-iso-base builders. See

image-builder-ova.py assumes the value of guestostype is a key of this map

OS_id_map = {"vmware-photon-64": {"id": "36", "version": "", "type": "vmwarePhoton64Guest"},
"centos7-64": {"id": "107", "version": "7", "type": "centos7_64Guest"},
"centos8-64": {"id": "107", "version": "8", "type": "centos8_64Guest"},
"rhel7-64": {"id": "80", "version": "7", "type": "rhel7_64Guest"},
"rhel8-64": {"id": "80", "version": "8", "type": "rhel8_64Guest"},
"rockylinux-64": {"id": "80", "version": "", "type": "rockylinux_64Guest"},
"ubuntu-64": {"id": "94", "version": "", "type": "ubuntu64Guest"},
"flatcar-64": {"id": "100", "version": "", "type": "otherLinux64Guest"},
"Windows2019Server-64": {"id": "112", "version": "", "type": "windows2019srv_64Guest"}}

Therefore, vsphereguestostype should not be like other3xLinux64Guest and it shouldn’t overwrite guestostype but it does.

No idea how it does work.

Erkan Erol

2023-11-06 17:21:01

Since the value in the OVF file of recent images is otherLinux64Guest , I guess the script takes the right argument somehow. Therefore, I believe we need this change

Yiyi Zhou

2023-11-06 17:43:59

Thank you for getting to the bottom of this issue!

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-06 17:45:46

I think it’d be good to eventually refactor that script so that we don’t have those hardcoded values if possible. @Erkan Erol any chance you could open a basic issue for us to look at that in the future?

Erkan Erol

2023-11-08 08:07:51

I just opened an issue I have some urgent things for my daily job so I don’t have time for this issue now. I can check it later.

GitHub

Improve code for vSphere metadata setting · Issue #1339 · kubernetes-sigs/image-builder

Intro: We have discussed metadata in the OVF file for Flatcar in the slack thread and then I opened a PR to fix the Flatcar case #1337 It is still being discussed since the desired output is not cl...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1339

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-08 13:14:23

Just having the issue for now is a big improvement. Thanks for taking the time 😁

voor

2023-10-31 14:09:02

Cross posting since this seems related to image builder:

Robert Van Voorhees (https://kubernetes.slack.com/team/U2B8TBG58)

We're having issues with cloud init 23.3.1, things were working with 23.2.2, has anybody experienced similar issues? Seems like it's not running properly, this is on capa ec2

Original URL: https://kubernetes.slack.com/archives/C8TSNPY4T/p1697637405978589

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-31 15:05:47

There was a post a little way up about this, and a linked issue:

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)

Is there anyone with an understanding / experience with cloud-init in Ubuntu that might be able to help out @Shalin Patel with this issues - <a href="https://github.com/kubernetes-sigs/image-builder/issues/1333">https://github.com/kubernetes-sigs/image-builder/issues/1333</a> ? It looks like something changed in the <code>23.3.1-0ubuntu1~20.04.1</code> release of cloud-init that broke builds for cluster-api-provider-aws. 😞

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1698390626499849

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-31 15:06:17

🙈 Which I see linked to your comment in the thread

Marcus Noble (k8s@marcusnoble.co.uk)

2023-10-31 15:09:38

@voor Would you mind updating with the latest outcome from your thread to include the things tried and ruled out? Would hate for it to get lost in the noise of Slack.

GitHub

Cloud-init fails for ubuntu 20.04 base AMI and Cloud-init version '23.3.1-0ubuntu1~20.04.1' · Issue #1333 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1333

Phil H

2023-11-03 16:31:22

@Phil H has joined the channel

libsysguy

2023-11-03 17:47:14

@libsysguy has left the channel

Jayesh

2023-11-06 08:32:58

Hi folks. Is there a provision by which I can build a RHEL kubernetes image for vsphere but instead of using an rhel iso as base, I can use a different format something like a vmdk?

Maarten

2023-11-06 10:06:35

@Maarten has joined the channel

Slackbot

2023-11-06 14:30:22

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2023-11-06 14:31:10

I'm away on holiday this week so won't make this.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-06 14:33:19

I’m at Kubecon and can’t make it

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-06 14:57:58

The agenda is empty and several people aren’t able to make it so let’s skip. 🙂

mboersma

2023-11-06 15:50:48

Kubecon week is always a tough time for community meetings, I agree let's skip until Nov. 20th.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-06 15:51:54

I’m currently sat in the contributor summit so I’ll try and convince more people to join us on image-builder! 😄

👍 mboersma, jsturtevant

😄 Erkan Erol

:megaman: jsturtevant

Jayesh

2023-11-07 04:58:20

Hi folks, Sorry for a follow up on this, any help would be appreciated.

Jayesh Srivastava (https://kubernetes.slack.com/team/U023B885W9M)

Hi folks. Is there a provision by which I can build a RHEL kubernetes image for vsphere but instead of using an rhel iso as base, I can use a different format something like a vmdk?

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1699259578059149

harrison_latimer

2023-11-08 14:48:48

@harrison_latimer has joined the channel

Jonas Pedersen

2023-11-12 15:19:21

@Jonas Pedersen has joined the channel

Jonas Pedersen

2023-11-12 15:24:28

Hello
Im trying to build and use an image for clusterApi to use to create k8s clusters with out direct internet connection. So Im using Nexus as a proxy. I have successfully created a image with version 1.28.3.
My challenge is that when I generate a cluster.yml for kubectl to apply the cluster that clusterApi creates still is trying to connect to registry.k8s.io. And that will not work in my setup. I cant seem to find where to change this kubeadm config yaml to use my nexus proxy.

Abhay Krishna Arunachalam

2023-11-12 17:10:49

Have you tried setting the field .spec.kubeadmConfigSpec.clusterConfiguration.imageRepository to your private registry?

Jonas Pedersen

2023-11-12 18:50:15

Yes I have configured the ansible template to point to my own repo. It is somehow sadly ignored or later overwritten

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-12 23:12:21

I’m currently travelling and on mobile so not got a link to hand but there was a very similar question a couple months ago in this channel that you might be able to find. Hopefully it contains the answer you need but I’m not 100% sure.

Abhay Krishna Arunachalam

2023-11-13 00:28:34

I think this is the thread Marcus is referring to.

https://kubernetes.slack.com/archives/C01E0Q35A8J/p1678141651934979

Jhonathan Cavalcante (https://kubernetes.slack.com/team/U02NR9XDWUT)

[:openstack: Built CAPI Openstack Image breaking in kubeadm init command]

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1678141651934979

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-16 11:17:48

I'm going to get a new release published so the latest fixes for vsphere are available to use. Speak up now if there's something you're also wanting to get included that isn't yet merged.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-16 11:31:22

Kicking off the process now. If there's some other changes we need we can always do another release 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-16 11:41:35

@mboersma I'm confused slightly about one step of the release process:

Images are built by the pr-container-image-build job. This will push the image to a staging repository.

The image does end up in the staging repository but I don't see any new job appear in testgrid for it. 😕 Any idea why not?

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-16 11:45:06

Waiting on image promotion PRs being approved:

GitHub

Image promotion for scl-image-builder v0.1.21 by AverageMarcus · Pull Request #6089 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.21 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers "@AverageMarcus @jsturtevant @kkes...

Original URL: https://github.com/kubernetes/k8s.io/pull/6089

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-16 13:54:49

Docs PR:

GitHub

Update docs after v0.1.21 release by AverageMarcus · Pull Request #1344 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1344

mboersma

2023-11-22 21:57:52

Hey @Marcus Noble, sorry I was out for a few days so didn't see this thread until now.

Yes, I don't see the jobs in testgrid either, although I've learned that the images will show up in staging anyway, I know the jobs did show up at one point after we nailed down the release process...I'm not sure what broke.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-22 21:59:27

The images are built as expected, I just cant find any logs / status showing them being built 🤷

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-16 13:48:01

Image-builder v0.1.21 is now available:
Thanks to all contributors! 💙 🎉

GitHub

Release v0.1.21 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.21">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.21</a> What's Changed Updated docs post v0.1.20 release by @Average...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.21

:thank_you: mboersma

🎉 mboersma, nikparasyr

abdulraheem

2023-11-18 06:16:41

@abdulraheem has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-20 07:59:33

The image-builder office hours are due later today but there is currently only one item on the agenda (the announcement of the new release just above this post). If you have any topics you'd like to discuss please add them to the agenda. If it's still empty by the time the automated reminder message gets posted in here I'll conclude we have nothing to discuss this week and cancel it for today. 🙂

Slackbot

2023-11-20 14:30:23

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-20 14:38:33

☝️ Agenda still empty so I'm going to cancel for today! 🙂

Drew Hudson-Viles

2023-11-20 14:38:44

No problem 🙂

Martin

2023-11-21 21:36:58

@Martin has joined the channel

Martin

2023-11-21 21:37:59

I am trying to use a ubuntu image for vmware from here I don’t understand how the auth part works I don’t see any default passwords posted? I am I missing something ?

Dharmjit

2023-11-23 10:43:53

I guess the published node images don't have passwords, ssh keys and its expected to be used with cluster-api, infra-providers which set ups the required credentials for ssh access.

2023-11-23 19:06:00

@FG has joined the channel

Slackbot

2023-11-25 17:05:02

This message was deleted.

Martin

2023-11-25 17:09:54

sorry about that posted to the wrong place, thanks

Marcus Noble (k8s@marcusnoble.co.uk)

2023-11-25 17:18:03

No worries 🙂

Slackbot

2023-12-04 14:30:13

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:16:14

☝️ The agenda is currently empty. Does anyone have anything they're like to discuss? If not I'm happy to skip but it's likely to be the last sync until next year.

mboersma

2023-12-04 15:45:55

I don't have any specific topics to discuss, but I can join if something gets added.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 16:22:01

Still no topics so I'm going to skip.

Happy holidays and hope you all enjoy the festive period! 🎉

👍 mboersma, jsturtevant

Drew Hudson-Viles

2023-12-04 15:17:05

I'm playing nurse to the family today so may not be able to join anyway I'm afraid! I've got some potential updates to nvidia bits incoming soon™ but that can wait until next year

:ack_tcp: Marcus Noble

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:42:52

I've also realised that since the DST change that reminder in here is an hour earlier than it should be 🙈

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:43:47

:thank_you: mboersma, Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:44:19

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 3:30PM every other Monday (next occurrence is December 11th), Greenwich Mean Time.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:44:50

🤦‍♂️ I now don't know how to get it to start the reminder in two weeks

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:45:16

set up a reminder “from 18th December at 3:30pm Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 9AM every Monday, Greenwich Mean Time.

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-04 15:45:58

Nevermind, I'm going to set myself a reminder to create a new channel reminder in the new year 😆

😆 Drew Hudson-Viles

😄 mboersma

jsturtevant

2023-12-07 22:57:35

⚠️ I've opend a PR to remove the docker implementation related code for Windows. It was used with dockershim which was removed in 1.24 in k/k and relied on a bug in Docker Windows implementation for the support. This means containerd will be default (and only supported runtime for windows) in image builds. I believe most Windows implementations were already using containerd so should not be an issue

GitHub

Remove windows docker by jsturtevant · Pull Request #1354 · kubernetes-sigs/image-builder

What this PR does / why we need it: Dockershim was completly removed in 1.24. The docker support that was added here was relied on docker shim. Which issue(s) this PR fixes (optional, in fixes #(,...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1354

Marcus Noble (k8s@marcusnoble.co.uk)

2023-12-08 07:40:32

relied on a bug

🙈

:nod: jsturtevant

Shahar Shavit

2023-12-12 16:04:01

@Shahar Shavit has joined the channel

John Payne

2023-12-12 22:27:11

@John Payne has joined the channel

John Payne

2023-12-12 22:29:04

Hi all. If I am using a fresh ubuntu VHD, what else should I add to ensure that it can be used within an azure cluster?

jsturtevant

2023-12-13 00:44:05

are you using CAPZ?

jsturtevant

2023-12-13 00:46:44

Using image builder, if you run build-azure-vhd-ubuntu-2204 (or version equivalent ) it will prepare everything needed to run with CAPZ. If you are interested in the details you can browse the ansible that configures it

Shashinandan Srinivasa

2023-12-13 02:14:27

Hello Folks, v0.1.21 does not remove machine ID (/etc/machine-id) while building vsphere template(build-node-ova-vsphere-ubuntu-2204).. Anyone facing similar issue, or am I doing something wrong? This is causing all VMs being assigned same IP address. Anyone else facing same issue. I can see in build logs that it shows truncated, but it really isn't

Shashinandan Srinivasa

2023-12-13 02:14:43

vsphere-iso.vsphere: TASK [sysprep : Truncate machine id] *
2251 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/machine-id', 'state': 'absent', 'mode': '0644'})
2252 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/machine-id', 'state': 'touch', 'mode': '0644'})
2253 vsphere-iso.vsphere:
2254 vsphere-iso.vsphere: TASK [sysprep : Truncate hostname file] *
2255 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/hostname', 'state': 'absent', 'mode': '0644'})
2256 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/hostname', 'state': 'touch', 'mode': '0644'})

Shashinandan Srinivasa

2023-12-13 03:27:14

confirmed that building template using image-builder version v0.1.19 works fine.. v0.1.21 has issues

mboersma

2023-12-13 20:09:43

@Shashinandan Srinivasa could you open a GitHub issue for that?

Shashinandan Srinivasa

2023-12-14 17:39:04

Sure, running some more builds to see where it is going wrong. I will raise an issue after.

:thank_you: mboersma

Hugo Prudente

2023-12-13 15:28:16

@Hugo Prudente has joined the channel

Hugo Prudente

2023-12-13 15:30:24

Hi Folks,

I have opened a PR to support Centos9 Stream, We are heavy users of CentOS on Openstack and we are on a project with K8S it, for that I had to build the Centos 9 image to be compliant with our security and I'm returning it to the community.

Thanks.

👍 mboersma

Gong (Grace) Zhang

2023-12-15 07:02:07

@Gong (Grace) Zhang has joined the channel

linkarzu

2023-12-16 10:47:01

@linkarzu has joined the channel

mboersma

2023-12-18 15:24:30

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2023-12-18 15:25:22

I see no agenda... Enjoy the holidays?

mboersma

2023-12-18 15:25:36

If you have something image-builder to discuss, or any question to ask, please add it to the agenda above (or just mention it here in this Slack thread.)
If we don't have any topics, we'll skip until 2024. 🙂

mboersma

2023-12-18 15:26:20

Yes probably @Drew Hudson-Viles but I'm available if anyone has something to discuss.

Drew Hudson-Viles

2023-12-18 15:26:41

I suspect a number of people may have already broken up for the break.

I'm also available too. I appear to be one of the few people in my company still working this week 😄

mboersma

2023-12-18 15:27:03

Same here. 😀

Drew Hudson-Viles

2023-12-18 15:29:29

tenor_gif5503164858707952577.gif

❤️ mboersma

mboersma

2023-12-18 16:01:35

I think you're right @Drew Hudson-Viles; looks like we should skip this meeting.
But two Mondays from now is New Year's Day... Should we wait three weeks and start up again on January 8th?

Drew Hudson-Viles

2023-12-18 16:01:51

That sounds good to me buddy.

mboersma

2023-12-18 16:02:29

Let's do it. I hope you have excellent holidays, see you soon!

:party_wizard: Drew Hudson-Viles

Drew Hudson-Viles

2023-12-18 16:03:15

I was going to suggest the week after as it's unlikely anything major will crop up... but you always expect that over the holidays and it always happens 😛

So yes, let's stick with the 8th.

Have a good break my friend and I'll speak to you after.

🎄 mboersma

mboersma

2023-12-18 16:03:33

We don't have any agenda topics, so let's skip today's office hours. Because two weeks from now is New Year's Day, we'll have the next one on January 8th, 2024.
Happy Holidays everyone!

mboersma

2024-01-08 12:40:00

I think actually the 15th is when the next meeting is scheduled, not today. Sorry about the misinfo.

👍 Drew Hudson-Viles

Thomas Güttler

2023-12-25 18:30:13

@Thomas Güttler has joined the channel

nawazkh

2023-12-29 21:58:49

@nawazkh has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-08 10:01:11

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 3:30PM every other Monday (next occurrence is January 15th), Greenwich Mean Time.

:thank_you: mboersma

Danny Seymour

2024-01-08 21:01:23

@Danny Seymour has joined the channel

mcbenjemaa

2024-01-09 17:23:01

@mcbenjemaa has joined the channel

mcbenjemaa

2024-01-11 12:05:40

Hello image-builder maintainers,

Recently, I talked to SCL Leads regarding a new proposal for a new sub-project,

A Project for automating image builds using the Kubernetes API (Similar to CAPI but for images)

They mentioned that this is potentially an image-builder evolution.

Therefore, I'm sharing this with you for discussion and brainstorming.

Find my design proposal:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-11 12:52:30

Hey @mcbenjemaa 👋 This is great! I love seeing new ideas like this!

I have some thoughts / questions...

If I'm not mistaken, this would be more a replacement for Packer rather than specifically image-builder, yes? The "Why" in the document seems to conflate the two and I don't think that's quite correct. For example, image-builder provides end users with a known good (ideally) set of configurations for building a VM image for a kubernetes node. With how Forge is describe in the doc, it looks like it would be on the end users to come up with and configure the build steps themselves as it would be the CRs that they apply.

Following on from the previous - do you foresee a way to package and ship pre-build Build CRs that could then be used by end users so they don't need to know exactly everything that is needed for a Kubernetes node (e.g. replacing what image-builder does)

What is the benefit of having this as an operator? I don't really understand what value having a reconciliation loop provides in this case. I see the building of a VM image as a one-time thing - you configure it and build the artifact.

Do you have any thoughts on how variables might be used to allow users to configure things like package versions? I suspect this is what you're hinting at with mentions of Helm and Kustomize but it would be could to have that spelled out in the proposal.

I'd like to know more about the issues you state people have reported with using image-builder - Image-builder is a bit lazy in updating and upgrading dependencies. and Users also have asked different questions about the usage.. I'm not aware of specifically what these are referring too so if you have any sources etc. it would be great for us so that we can address them and improve things in this project.

Have you thought about how this might fit into a CI/CD pipeline? The current operator-based approach doesn't really lend itself well to that model. Specifically I'd like to see things like "Triggers" (I currently run image-builder automatically whenever there is a new GitHub release in kubernetes/kubernetes) , "Validation / Tests" (in image-builder we use goss to validate the state and at GS we're aiming to automatically run both our in-house E2E test suite and the Kubernetes conformance tests for every VM image we build) and "Post-build steps" (for example, replicating an AMI to multiple AZs and marking it as public or something like copying a built disk image to one or more S3 buckets).

Templating is not possible for packer templates nor ansible playbooks. - this isn't correct. Packer with HCL allows for templating.

Several of the CAPI provider teams run currently run image-builder nightly using Prow to test their applications / pre-test new releases. Have you thought about how Forge might handle that requirement? (I think this is a similar question to the CI/CD pipeline one)

Hopefully that's not too overwhelming 😆 I'm really excited to see how this goes!

Also... I'm shocked that there isn't already a CNCF / Kubernetes project called "Forge"! 😮

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-11 13:07:16

Also, we have the next image-builder office hours on Monday - do you think you'd be able to join and present this there too?

mcbenjemaa

2024-01-11 13:33:28

Yes, i will join.

👍 mboersma, Drew Hudson-Viles

🎉 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-11 13:40:49

Awesome! 😄 i'll add it to the agenda then.

mcbenjemaa

2024-01-11 15:02:18

We are collecting thoughts and feedback.

The naming idea comes from a metaphor,
Forging Images
While the Core is an anvil, the infrastructure part is a blacksmith, and the provisioners are tools, like a Hammer.

First of All, as SCL leads stated, this could be an image-builder evolution like an image-builder v2.
However, Rebranding is a proposal whether it's approved or not.

Let me answer some of your questions:

If I'm not mistaken, this would be more a replacement for Packer rather than specifically image-builder, yes? The "Why" in the document seems to conflate the two and I don't think that's quite correct. For example, image-builder provides end users with a known good (ideally) set of configurations for building a VM image for a kubernetes node. With how Forge is describe in the doc, it looks like it would be on the end users to come up with and configure the build steps themselves as it would be the CRs that they apply.

This actually means a replacement for image-builder itself, and yes, this will get rid of Packer, but it still uses Ansible playbooks.
The aim here is to build the project and provide end-users with built-in templates so they can use them to build their images.
The proposal also mentions a CLI that can be used to help people migrate from packer/ansible to the Forge approach,
Which also means a migration plans for image-builder templates.

Following on from the previous - do you foresee a way to package and ship pre-build Build CRs that could then be used by end users so they don't need to know exactly everything that is needed for a Kubernetes node (e.g. replacing what image-builder does)

Yes, that's exactly what I was saying above. The image-builder will still have built-in templates or manifests. so people can use it.
With also a way of generating OR converting existing packer templates into BUILD CRs using the CLI.

What is the benefit of having this as an operator? I don't really understand what value having a reconciliation loop provides in this case. I see the building of a VM image as a one-time thing - you configure it and build the artifact.

this is summarized by the difference between the imperative and declarative approaches,
while the reconciliation loops are beneficial in terms of error detection and provide idempotent process,
However, there are also many out-of-the-box benefits of using CRDs, like Gitops, RBAC, Templating, etc.

Do you have any thoughts on how variables might be used to allow users to configure things like package versions? I suspect this is what you're hinting at with mentions of Helm and Kustomize but it would be could to have that spelled out in the proposal.

Well, many solutions could be used, like templating with kustomize or helm,
but also envsubst similar to CAPI cluster templates.

I'd like to know more about the issues you state people have reported with using image-builder - Image-builder is a bit lazy in updating and upgrading dependencies. and Users also have asked different questions about the usage.. I'm not aware of specifically what these are referring too so if you have any sources etc. it would be great for us so that we can address them and improve things in this project.

The word should be busy, upgrades are not very active.
The usage questions are teams internals with how to set up an automated pipeline to run image-builder and can build the same images for different regions, for example.

Have you thought about how this might fit into a CI/CD pipeline? The current operator-based approach doesn't really lend itself well to that model. Specifically I'd like to see things like "Triggers" (I currently run image-builder automatically whenever there is a new GitHub release in kubernetes/kubernetes) , "Validation / Tests" (in image-builder we use goss to validate the state and at GS we're aiming to automatically run both our in-house E2E test suite and the Kubernetes conformance tests for every VM image we build) and "Post-build steps" (for example, replicating an AMI to multiple AZs and marking it as public or something like copying a built disk image to one or more S3 buckets).

Actually, this is interesting.
I think the Build CRs should also be able to provide the image for multiple Regions.

Templating is not possible for packer templates nor ansible playbooks. - this isn't correct. Packer with HCL allows for templating.

This is actually should be re-phrase it,
Here I mean that we could not use features like a parent template, or bases such as with kustomize.

Several of the CAPI provider teams run currently run image-builder nightly using Prow to test their applications / pre-test new releases. Have you thought about how Forge might handle that requirement? (I think this is a similar question to the CI/CD pipeline one)

this is also an interesting point.
but can be solved by an e2e framework, which makes it possible to run using Prow.

jwest

2024-01-11 19:20:30

@jwest has joined the channel

Anurag

2024-01-12 14:40:26

@Anurag has joined the channel

Karine Santos

2024-01-12 17:45:08

@Karine Santos has joined the channel

Karine Santos

2024-01-12 17:48:29

hello everyone!
When trying to upload 2 images from k8s versions (1.26.12 and 1.28.5), I receive this error:

openstack: fatal: [default]: FAILED! => {"cache_update_time": 1705078887, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubelet=1.26.12-00' 'kubeadm=1.26.12-00' 'kubectl=1.26.12-00' 'kubernetes-cni=1.2.0-00'' failed: E: Version '1.26.12-00' for 'kubelet' was not found\nE: Version '1.26.12-00' for 'kubeadm' was not found\nE: Version '1.26.12-00' for 'kubectl' was not found\n", "rc": 100, "stderr": "E: Version '1.26.12-00' for 'kubelet' was not found\nE: Version '1.26.12-00' for 'kubeadm' was not found\nE: Version '1.26.12-00' for 'kubectl' was not found\n", "stderr_lines": ["E: Version '1.26.12-00' for 'kubelet' was not found", "E: Version '1.26.12-00' for 'kubeadm' was not found", "E: Version '1.26.12-00' for 'kubectl' was not found"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nPackage kubeadm is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\nPackage kubelet is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\nPackage kubectl is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package kubeadm is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", "", "Package kubelet is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", "", "Package kubectl is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}
5732    openstack:
5733    openstack: PLAY RECAP *
5734    openstack: default                    : ok=44   changed=34   unreachable=0    failed=1    skipped=190  rescued=0    ignored=0
5735    openstack:
5736==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
5737==> openstack: Deleted temporary floating IP 'e89775ba-1edc-4d47-821a-bfc94582f064' (209.127.141.191)
5738==> openstack: Terminating the source server: 50ce0061-5572-4b00-b052-de0422138e0f ...
5739==> openstack: Deleting volume: d7d0b4c0-f238-4cc4-96f0-25fcd85bbb50 ...
5740==> openstack: Deleting temporary keypair: packer_65a16bce-18a3-3179-db2a-328434dae677 ...
5741Build 'openstack' errored after 19 minutes 48 seconds: Error executing Ansible: Non-zero exit status: exit status 2
5742==> Wait completed after 19 minutes 48 seconds
5743==> Some builds didn't complete successfully and had errors:
5744--> openstack: Error executing Ansible: Non-zero exit status: exit status 2
5745==> Builds finished but no artifacts were created.
5746make[1]: * [Makefile:500: build-openstack-ubuntu-2204] Error 1
5747make[1]: Leaving directory '/builds/magalu-cloud-iaas/k8s/capi-image-builder'
5748+ clean_up_images
5749+ echo 'Starting image clean up'
5750Starting image clean up
5751+ grep -q -v -e '^$' ./images_list.tmp
5752+ echo 'No images to delete'
5753No images to delete

Do you know what it could be?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-12 19:07:14

I’m currently not near my laptop so can’t double check but I suspect it might be related to the Kubernetes package repo change.
What version of image-builder are you using and what does your vars look like?

Drew Hudson-Viles

2024-01-12 19:10:46

I was about to suggest the same. I think it is related to the repos and that 1.26 isn't available in the current ones. Can you try a build with 1.27 just to check? I suspect that will work.

Karine Santos

2024-01-12 20:30:37

Hello @Marcus Noble and @Drew Hudson-Viles !!!
Today we use a clone of image-builder and I believe it is not that up to date.
Can you help me where the version of the image builder and vars are defined in this repo?

GitHub

GitHub - kubernetes-sigs/image-builder: Tools for building Kubernetes disk images

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder

Drew Hudson-Viles

2024-01-12 20:37:39

Hi,

So the releases page will show you the latest versions available however one quick check you can do is to check the value of the repo you have defined in your fork - see this PR for the change that went in to update this and the file in which you can check.

This was updated back in September to reflect the new repos as the old ones were frozen and were expected to be removed in January of this year

Karine Santos

2024-01-15 09:38:00

Hello @Marcus Noble and @Drew Hudson-Viles !!!

@Drew Hudson-Viles For version 1.27 I get the error:

ERROR: python-cinderclient 9.4.0 has requirement requests>=2.25.1, but you'll have requests 2.22.0 which is incompatible.
3682Installing collected packages: pbr, stevedore, wcwidth, PrettyTable, zipp, importlib-metadata, autopage, pyperclip, attrs, cmd2, cliff, iso8601, oslo.i18n, netifaces, packaging, pytz, tzdata, pyparsing, wrapt, debtcollector, oslo.utils, msgpack, oslo.serialization, os-service-types, keystoneauth1, python-novaclient, rfc3986, oslo.config, python-keystoneclient, requestsexceptions, platformdirs, decorator, typing-extensions, dogpile.cache, jsonpointer, jsonpatch, openstacksdk, osc-lib, python-cinderclient, python-openstackclient
3683Successfully installed PrettyTable-3.9.0 attrs-23.2.0 autopage-0.5.2 cliff-4.5.0 cmd2-2.4.3 debtcollector-2.5.0 decorator-5.1.1 dogpile.cache-1.3.0 importlib-metadata-7.0.1 iso8601-2.1.0 jsonpatch-1.33 jsonpointer-2.4 keystoneauth1-5.5.0 msgpack-1.0.7 netifaces-0.11.0 openstacksdk-2.1.0 os-service-types-1.7.0 osc-lib-3.0.0 oslo.config-9.3.0 oslo.i18n-6.2.0 oslo.serialization-5.3.0 oslo.utils-6.3.0 packaging-23.2 pbr-6.0.0 platformdirs-4.1.0 pyparsing-3.1.1 pyperclip-1.8.2 python-cinderclient-9.4.0 python-keystoneclient-5.3.0 python-novaclient-18.4.0 python-openstackclient-6.4.0 pytz-2023.3.post1 requestsexceptions-1.4.0 rfc3986-2.0.0 stevedore-5.1.0 typing-extensions-4.9.0 tzdata-2023.4 wcwidth-0.2.13 wrapt-1.16.0 zipp-3.17.0

@Marcus Noble for my repo:
cni.json:

{
  "kubernetes_cni_deb_version": "1.2.0-00",
  "kubernetes_cni_http_checksum": "sha256: kubernetes&#95;cni&#95;http&#95;checksum&#95;arch}}-v1.2.0.tgz.sha256",
  "kubernetes_cni_http_checksum_arch": "amd64",
  "kubernetes_cni_http_source": "",
  "kubernetes_cni_rpm_version": "1.2.0-0",
  "kubernetes_cni_semver": "v1.2.0",
  "kubernetes_cni_source_type": "pkg"
}

packerkubernetes.json :
{
"crictl_arch": "amd64",
"crictl_sha256": " crictl_version}}/crictl-v{{user crictl_version}}-linux-{{user crictl_arch}}.tar.gz.sha256",
"crictl_source_type": "pkg",
"crictl_url": " crictl_version}}/crictl-v{{user crictl_version}}-linux-{{user crictl_arch}}.tar.gz",
"crictl_version": "{{env CRICTL_VERSION}}",
"kubeadm_template": "etc/kubeadm.yml",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_deb_gpg_key": "",
"kubernetes_deb_repo": "\" kubernetes-xenial\"",
"kubernetes_deb_version": "{{env KUBE_VERSION}}-00",
"kubernetes_http_source": "",
"kubernetes_load_additional_imgs": "false",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_rpm_gpg_key": "\" \"",
"kubernetes_rpm_repo": " kubernetes_rpm_repo_arch}}",
"kubernetes_rpm_repo_arch": "x86_64",
"kubernetes_rpm_version": "{{env KUBE_VERSION}}-0",
"kubernetes_semver": "v{{env KUBE_VERSION}}",
"kubernetes_series": "v{{env KUBE_SERIES}}",
"kubernetes_source_type": "pkg",
"systemd_prefix": "/usr/lib/systemd",
"sysusr_prefix": "/usr",
"sysusrlocal_prefix": "/usr/local"
}
for my variables:
CRICTLVERSION: "1.26.1"
KUBEVERSION : "1.26.12" and "1.28.5"
KUBESERIES : "1.26" and "1.28"

Drew Hudson-Viles

2024-01-15 09:43:50

So it does look like being out of date is an issue here.

for 1.27 I can see you're getting ERROR: python-cinderclient 9.4.0 has requirement requests>=2.25.1, but you'll have requests 2.22.0 which is incompatible.

For the other info you've provided you are using

 "kubernetes_deb_repo": "\" kubernetes-xenial\"",

Where as it should now be

"kubernetes_deb_repo": " user kubernetes&#95;series }}/deb/",

I think as it's a fork you're options here are to manually update the fields or sync the fork with the main image-builder repo as it appears it is out of date.

Unless Marcus can suggest anything else of course! 😉

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 09:47:32

That sounds sensible to me.

Karine Santos

2024-01-15 12:26:55

I changed the:

"kubernetes_deb_repo": " user kubernetes&#95;series }}/deb/",

and also the ansible_args.json:

openstack: fatal: [default]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/home/ubuntu/~core/.ansible/tmp/ansible-tmp-1705320189.7624567-59973788767630/AnsiballZ_apt_repository.py\", line 102, in \n    _ansiballz_main()\n  File \"/home/ubuntu/~core/.ansible/tmp/ansible-tmp-1705320189.7624567-59973788767630/AnsiballZ_apt_repository.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/ubuntu/~core/.ansible/tmp/ansible-tmp-1705320189.7624567-59973788767630/AnsiballZ_apt_repository.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.packaging.os.apt_repository', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_7ka7dvjo/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in \n  File \"/tmp/ansible_apt_repository_payload_7ka7dvjo/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 547, in main\n  File \"/usr/lib/python3/dist-packages/apt/cache.py\", line 152, in __init__\n    self.open(progress)\n  File \"/usr/lib/python3/dist-packages/apt/cache.py\", line 214, in open\n    self._cache = apt_pkg.Cache(progress)\napt_pkg.Error: E:Malformed entry 1 in list file /etc/apt/sources.list.d/kubernetes.list (Component), E:The list of sources could not be read.\nConnection to 127.0.0.1 closed.\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Karine Santos

2024-01-15 14:19:49

Any suggestions as to what it could be @Marcus Noble @Drew Hudson-Viles?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 14:22:05

Connection to 127.0.0.1 closed. That sounds like its possibly network error. Does this consistently fail the same if you re-run it?

Drew Hudson-Viles

2024-01-15 14:25:09

To add to that you also have E:Malformed entry 1 in list file /etc/apt/sources.list.d/kubernetes.list (Component), E:The list of sources could not be read and so I suspect many values have been changed since you've last synced with the project and that changing those few lines would be enough.

I'd recommend bringing the latest changes into your project via a sync with the upstream and then if you have any custom requirements on top of that, ensure they are still valid and work with where the project is at now.

If you currently have changes in your main branch, the easiest approach would be to check them out into another branch so that you don't lose anything, sync with upstream and then PR those changes back in.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 14:26:57

Alternatively, if you don't have any code changes in your fork and only changes to vars I'd recommend using the container image that we build for each new release. Then you can be sure that the versions of binaries used are ok too.

👍 Drew Hudson-Viles

Karine Santos

2024-01-16 13:55:22

Hello @Drew Hudson-Viles and @Marcus Noble!!!
Thank you for your help! I followed exactly these updates here and managed to upload my new images.

GitHub

Install Kubernetes artifacts from pkgs.k8s.io · kubernetes-sigs/image-builder@5761d5e

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/commit/5761d5e6026938a71584c7c2672ae718436f211d

🙌 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-16 14:58:57

🎉 That mean you were able to successfully build the images? 😄

mcbenjemaa

2024-01-15 15:07:23

I'm trying to build QEMU images with image-builder from a GitHub action job,
But I got this error:

The builder qemu is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.



Error: Failed to initialize build "qemu"

Shouldn;t the packer plugin for QEMU is part of deps-qemu ?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:12:25

I'm guessing you're on a newer version of Packer than is supported by image-builder as the qemu plugin was built-in until v1.10.0.
Are you using the provided image-builder container image? If so I suspect we have something wrong in the deps scripts.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:14:05

v1.9.5 of Packer is the latest supported by image-builder due to the licence change.

👍:skin_tone_3: mcbenjemaa

mcbenjemaa

2024-01-15 15:23:33

yeah, Packer 1.10.0

mcbenjemaa

2024-01-15 15:24:43

I can use the container image?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:25:17

You should be able to. I've never done it in a GitHub action but I don't see why it wouldn't work.

mcbenjemaa

2024-01-15 15:27:42

, yeah i will try it. Thanks

mcbenjemaa

2024-01-15 15:28:06

btw, is the office hours 16:30 Or 17:30?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:29:13

16:30 UTC

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:29:45

There should be a reminder message posted in 1 minute 😛

😁 mcbenjemaa

mcbenjemaa

2024-01-15 15:32:59

yeah, starts in 1 hour.

mcbenjemaa

2024-01-15 15:34:45

docker run --name image-builder <br> registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.21 make build-qemu-ubuntu-2204

is the container run, something like that?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:35:51

Yeah. You'll need to mount in the needed vars files and maybe some env vars, depending on how you're configuring, but that's pretty much it

mcbenjemaa

2024-01-15 15:36:32

right.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 07:31:43

We've had an issue opened about this version problem:
I've created a new issue to update image-builder to restrict the versions of Packer is will run with:

mcbenjemaa

2024-01-19 13:34:35

Good initiative

mcbenjemaa

2024-01-19 13:38:25

I have an update regarding the Forge project proposal:

The SCL leads are not very keen, that both projects coexists.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 13:40:23

What was their reasoning?

mcbenjemaa

2024-01-19 14:39:38

The fact that people need time to migrate is acceptable, but in order to make it happen there should be some sort of strong intent leading to feature parity in a reasonable time and to a plan with a sustainable deadline for the phase off, and unfortunately it seems to me we are missing both (intent and feature parity expectation)
I know that gaining consensus takes time, but now this smells like “yeah, nice call, but ultimately they will co-exist indefinitely” which is something I’m not really happy about

mcbenjemaa

2024-01-19 14:40:44

I'm looking now for consensus within the community.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 14:45:30

I know that gaining consensus takes time, but now this smells like “yeah, nice call, but ultimately they will co-exist indefinitely” which is something I’m not really happy about

I disagree with this. I think having options that suit different needs is totally fine. Just look at all the different ways of creating clusters and the different ways of running Kubernetes. There is no one solution for everyone. 🤷

mboersma

2024-01-19 20:18:10

Yes, I thought we had the idea of experimental or incubator projects to accommodate things like that.
I agree overall that if we want to replace image-builder someday forge has to aim for feature parity, which seems like a lot to do. But I don't think there would be a huge amount of resistance to deprecating image-builder in favor of forge if it met the same requirements.

👍:skin_tone_3: mcbenjemaa

Slackbot

2024-01-15 15:30:30

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-15 15:40:55

☝️ In just under an hour we'll have @mcbenjemaa giving an introduction to Forge that he shared a few days ago. Everyone is welcome to join. If anyone has any other discussion topics please add them to the agenda or let me know. 🙂 See y'all soon.

Mohamed chiheb ben jemaa (https://kubernetes.slack.com/team/UF111SQ4U)

Hello image-builder maintainers, Recently, I talked to SCL Leads regarding a new proposal for a new sub-project, <code>A Project for automating image builds using the Kubernetes API (Similar to CAPI but for images)</code> They mentioned that this is potentially an image-builder evolution. Therefore, I'm sharing this with you for discussion and brainstorming. Find my design proposal: <a href="https://docs.google.com/document/d/1MTXug2CDfwSu0qmFCyohrAT8Cji1-20kZ1RzY_jCNlw/edit?usp=sharing">https://docs.google.com/document/d/1MTXug2CDfwSu0qmFCyohrAT8Cji1-20kZ1RzY_jCNlw/edit?usp=sharing</a>

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1704974740857049

👍 Drew Hudson-Viles, mboersma

:fire_blue: mcbenjemaa

mboersma

2024-01-15 18:12:05

A recording of today's image-builder office hours is available on YouTube:

mboersma

2024-01-16 14:48:29

Image-builder v0.1.22 is now available:
Thanks to all contributors! 💙 🎉

🙌 Karine Santos

:parrotk8s: Marcus Noble, mloskot, Drew Hudson-Viles, jsturtevant, Magnus RC, Anurag

🙌:skin_tone_2: Anurag

:_thank_you_: Anurag

:blod_tada: Anurag

Abhay Krishna Arunachalam

2024-01-19 07:35:44

This is an issue opened a month ago, and now we're facing it too. Could someone take a look? I have added a comment with some of my findings.

GitHub

TASK [sysprep : Reset network interface IDs] failed when building RHEL 9 image · Issue #1367 · kubernetes-sigs/image-builder

RHEL 9 image build fails with following error: TASK [sysprep : Reset network interface IDs] *********************************** fatal: [default]: FAILED! => {"changed": true, "cmd": "sed -i '/^\\(u...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1367

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 07:49:07

If I understand your comment it looks like you've found the solution, yes? Would you be willing to open a PR with the change to using nmcli?

Abhay Krishna Arunachalam

2024-01-19 08:35:35

Thanks for your time reading my comment. Here you go

GitHub

Migrate interface configuration files to NetworkManager keyfiles on RHEL 9 by abhay-krishna · Pull Request #1380 · kubernetes-sigs/image-builder

I opened this PR based on the findings in #1367 (comment). Fixes #1367.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1380

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 08:36:01

Wow that was quicker than I expected 😆 Thanks!

Abhay Krishna Arunachalam

2024-01-19 08:36:17

I kept the PR description terse as I have referenced my comment, hope that's okay

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 08:37:07

Is the Reset network interface IDs task still needed after this change?

Abhay Krishna Arunachalam

2024-01-19 08:43:42

I believe so because this change doesn't affect the behavior of that task, it just puts the nmconnection file in place so that the sed command can find it and delete the uuid.

Here are the contents of the file before the change, the Reset network interface IDs removes the uuid

Screenshot 2024-01-19 at 12.39.14 AM.png

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 08:44:19

Gotcha! So it makes sure that dir is setup and then the next task is able to do its thing 🙂 👍

Abhay Krishna Arunachalam

2024-01-19 08:45:17

exactly, I saw that even without the nmcli change, the /etc/NetworkManager/system-connections dir is present but it's empty as the machine was still using the ifcfg files

Abhay Krishna Arunachalam

2024-01-19 08:45:35

Thanks for the lgtm! 🚀

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-19 08:45:45

I've added my lgtm. I'll leave it for one of the other maintainers to add theirs too once the tests pass if thats cool with you 🙂

Abhay Krishna Arunachalam

2024-01-19 08:47:17

Yeah that sounds good!

👍 Marcus Noble

Yongxiang Gao

2024-01-22 22:41:29

@Yongxiang Gao has joined the channel

Yongxiang Gao

2024-01-22 22:44:45

On an Azure ubuntu VM node, I try to build ubuntu image 2004 for bare metal (raw) with full disk encryption support, however, the build stuck here:

==> qemu: Retrieving ISO
==> qemu: Trying 
==> qemu: Trying 
==> qemu:  => /root/.cache/packer/48e4ec4daa32571605576c5566f486133ecc271f.iso
==> qemu: Starting HTTP server on port 8529
==> qemu: Found port for communicator (SSH, WinRM, etc): 3319.
==> qemu: Looking for available port between 5900 and 6000 on 127.0.0.1
==> qemu: Starting VM, booting from CD-ROM
  qemu: The VM will be run headless, without a GUI. If you want to
  qemu: view the screen of the VM, connect via VNC without a password to
  qemu: 
==> qemu: Waiting 10s for boot...
==> qemu: Connecting to VM via VNC (127.0.0.1:5986)
==> qemu: Typing the boot commands over VNC...
  qemu: Not using a NetBridge -- skipping StepWaitGuestAddress
==> qemu: Using SSH communicator to connect: 127.0.0.1
==> qemu: Waiting for SSH to become available...

How to trouble shoot, is there something like remote/web console in vcode of vSphere build to check the VM status?

Yongxiang Gao

2024-01-23 02:41:30

Look like there are following kvm process running:

root   3807270 3807149 3 00:17 pts/0  00:02:03 /usr/bin/qemu-system-x86_64 -smp 1 -drive if=none,file=output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7,id=drive0,cache=writeback,discard=unmap,format=raw -drive file=/root/.cache/packer/48e4ec4daa32571605576c5566f486133ecc271f.iso,media=cdrom -boot once=d -vnc 127.0.0.1:15 -m 2048M -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -machine type=pc,accel=kvm -bios OVMF.fd -name ubuntu-2004-kube-v1.26.7 -netdev user,id=user.0,hostfwd=tcp::3837_:22
root   3807273    2 0 00:17 ?    00:00:00 [kvm-nx-lpage-re]
root   3807276    2 0 00:17 ?    00:00:00 [kvm-pit/3807270]

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-23 06:45:34

I haven't seen this issue before but also not seen image-builder run in Azure -> vSphere myself so not 100% sure if you've hit a known bug or not.

Just to confirm, are you using the latest release of image-builder?

The Waiting for SSH to become available... makes me suspect it might be some form of firewall blocking access. Are you able to confirm that port 22 should be open between Azure and vSphere?

Yongxiang Gao

2024-01-23 09:25:32

Yes, I use the latest image-builder source code plus the patches from eks-anywhere-build-tooling for image-builder.

The issue maybe related to encryption change I put in preseed-efi.cfg, the process chain is like this (parent down to child):

make build-raw-ubuntu-2004-efi

packer build ...

packer build ...

packer plugin packer-builder-qemu

qemu-system-x8664 -smp 1 -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -name ubuntu-2004-kube-v1.26.7 -machine type=pc,accel=kvm -vnc 127.0.0.1:89 -m 2048M -netdev user,id=user.0,hostfwd=tcp::3223:22 -bios OVMF.fd -drive if=none,file=output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7,id=drive0,cache=writeback,discard=unmap,format=raw -drive file=/root/.cache/packer/48e4ec4daa32571605576c5566f486133ecc271f.iso,media=cdrom -boot once=d

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-23 09:28:12

patches from eks-anywhere-build-tooling for image-builder.

What are these? I've never come across these before 😮

Based on what you say it does sound like the encryption might be related. Are you able to test the same config without the encryption to see if that succeeds?

Yongxiang Gao

2024-01-23 09:47:33

The fix some issues and some improvements, make it EKSA conformant.

Yongxiang Gao

2024-01-23 09:48:13

Right, without the encryption related changes, the build pass

Yongxiang Gao

2024-01-23 09:49:02

Do you know someone(s) have working solution to support full-disk encryption?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-23 09:49:47

Not that I've seen. Doesn't look like there's any related issues either.

Yongxiang Gao

2024-01-23 09:52:31

I know we can switch to full disk encryption when install ubuntu on individual machine manually.
However, I would like to do it automatically to create EKS anywhere cluster on bare metal.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-23 09:54:55

Yeah I get that. Not sure what is currently blocking it. I would think that Packer would be possible with full disk encryption but perhaps there's some configuration or flag that we're currently missing.

Yongxiang Gao

2024-01-23 17:43:23

How can I get familiar with packer and qemu-system-x86_64 quickly?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-23 17:45:42

For Packer the docs cover quite a lot of stuff if you search for things -
For qemu - I have no idea. I know next to nothing about it myself. 😞

:ty: Yongxiang Gao

Drew Hudson-Viles

2024-01-31 16:11:28

I've just tried this locally as I'm testing a build for KubeVirt and it looks like on my side it's getting stuck at the Select Language screen. I'll see if I can spot why.

Drew Hudson-Viles

2024-01-31 16:12:13

Also ,this is just QMEU w/22.04-efi but I suspect something similar is happening as it's hanging at the Waiting for SSH to become available prompt.

Drew Hudson-Viles

2024-01-31 16:25:11

ok so in my case I thought I'd fat fingered a config but it seems on reboot it's not ejecting the "cdrom" and so it's booting back into the installer. This prevents the next phase from running.

Lemmie see if I can solve this.

Drew Hudson-Viles

2024-01-31 17:14:25

This is funky - it seems it's working now and I've all I've done so far is increased the disk size double_facepalm . The efibootorder is obvioulsy doing it's job but I wonder if disk space was an issue with the default of 2G... Not certain tbh but it's all I've changed.

Maybe I'm not cut out for this computer stuff 😄

👍 Yongxiang Gao

Yongxiang Gao

2024-02-01 17:50:49

Please share your changes in detail. ty

Drew Hudson-Viles

2024-02-01 18:06:36

Hi, I've sent you a DM with the details but in case anyone else comes across this, my changes are in this PR.

https://github.com/kubernetes-sigs/image-builder/pull/1389

GitHub

patch: qemu efi fixes by drew-viles · Pull Request #1389 · kubernetes-sigs/image-builder

What this PR does / why we need it: Set partition size for root to be -1 to prevent issues with disk sizes when someone wants to use the "disk_size" parameter in packer. Set the efibootmgr to look...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1389

Jasper

2024-01-24 09:24:09

@Jasper has joined the channel

Yongxiang Gao

2024-01-25 18:31:51

Where (the source code) are these messages from?
# make build-raw-ubuntu-2004-efi
....

==> qemu: Retrieving ISO
==> qemu: Trying 
==> qemu: Trying

Abhay Krishna Arunachalam

2024-01-25 18:37:51

👍 Yongxiang Gao

Yongxiang Gao

2024-01-25 22:42:29

@Abhay Krishna Arunachalam I try to support full disk encryption with change in preseed-efi.cfg as follows:

diff --git a/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg b/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
index 14cb4008f..fca87df75 100644
--- a/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
+++ b/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
@@ -52,7 +52,12 @@ d-i partman-partitioning/default_label string gpt
 d-i partman/choose_label string gpt
 d-i partman/default_label string gpt
 
-d-i partman-auto/method string regular
+#d-i partman-auto/method string regular
+d-i partman-auto/method string crypto
+d-i partman-crypto/confirm boolean true
+d-i partman-crypto/method string luks
+d-i partman-crypto/passphrase password possible
+d-i partman-crypto/passphrase-again password possible
 d-i partman-auto/choose_recipe select gpt-boot-root-swap
 d-i partman-auto/expert_recipe string     <br>  gpt-boot-root-swap ::           <br>@@ -78,6 +83,8 @@ d-i partman/choose_partition select finish
 d-i partman/confirm boolean true
 d-i partman/confirm_nooverwrite boolean true
 
+d-i initramfs-tools/cryptroot-initramfs-tools/verbose boolean true
+
 # Create the default user.
 d-i passwd/user-fullname string builder
 d-i passwd/username string builder
@@ -93,6 +100,9 @@ d-i grub-installer/with_other_os boolean true
 d-i finish-install/reboot_in_progress note
 d-i pkgsel/update-policy select none
 
+d-i debian-installer/add-kernel-opts string <br>+  "cryptopts=target=root,source=/dev/sda3,luks"

The build stuck at waiting for SSH:

==> qemu: Connecting to VM via VNC (127.0.0.1:5975)
==> qemu: Typing the boot commands over VNC...
  qemu: Not using a NetBridge -- skipping StepWaitGuestAddress
==> qemu: Using SSH communicator to connect: 127.0.0.1
==> qemu: Waiting for SSH to become available...

Am I missing something?

Abhay Krishna Arunachalam

2024-01-25 22:43:28

Did you not face this error before these changes?

Yongxiang Gao

2024-01-25 23:48:25

Right, the build passes if I don't put the changes there

Yongxiang Gao

2024-01-27 08:37:23

Something wrong with packer-plugin-qemu builder, especially at stepTypeBootCommand

Michi Altstaedt

2024-01-26 09:38:26

@Michi Altstaedt has joined the channel

Michi Altstaedt

2024-01-26 09:55:19

[k8s 1.29.1] which versions i must put in json for kubernetescnideb_version? I fail to determine what is available and what is compatible to this k8s version.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-26 10:36:20

I think you will want something like this:

{
  "kubernetes_deb_version": "1.29.1-00",
  "kubernetes_rpm_version": "1.29.1-0",
  "kubernetes_semver": "v1.29.1",
  "kubernetes_series": "v1.29",
}

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-26 10:36:39

Those were what I used to built from Flatcar on AWS.

Anurag

2024-01-26 11:10:22

If you're using pkgs.k8s.io then this might be helpful to you.

In general, I was not able to find a page where it's written which CNI version to use with an upstream kubernetes release.
Matt suggested to use a container and find out and I think you'll have to do the same. I like/use this same approach.

Having said that, with 1.29 series, you'll use 1.3.0 one as kubernetescnideb_version

Michi Altstaedt

2024-01-26 11:15:21

thanks @Anurag, that worked for me as well. I used these values:

    "kubernetes_cni_deb_version": "1.3.0-1.1",
    "kubernetes_cni_http_checksum": "sha256: kubernetes&#95;cni&#95;http&#95;checksum&#95;arch}}-v1.3.0.tgz.sha256",
    "kubernetes_cni_http_checksum_arch": "amd64",
    "kubernetes_cni_http_source": "",
    "kubernetes_cni_rpm_version": "1.3.0",
    "kubernetes_cni_semver": "v1.3.0",
    "kubernetes_cni_source_type": "pkg",

:blod_tada: Anurag

mboersma

2024-01-29 15:51:32

I'm trying to work out a more general solution to this (see bug #1363) but no luck yet. Ideas are appreciated!
The only change I know I'd like to make is to stop specifying the kubernetescnideb_version and just allow apt to figure out which version goes with the version of Kubernetes we're currently installing. (The user could still override that to choose a specific CNI package.) But I haven't got something similar working for rpms, and it doesn't help the "install from source" path that we also support for k8s+cni.

Slackbot

2024-01-29 15:30:16

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-29 15:31:45

☝️ The agenda is currently empty. Does anyone have anything they've like to raise? If not I'm in favour of skipping this week.

mcbenjemaa

2024-01-29 15:41:39

I just want to say that I will start a prototype for the Forge project with 2 volunteers before proposing to SCL.

And if anyone is interested in getting involved in the prototype, he is welcome to join.

nothing more 😄

👍 mboersma

mboersma

2024-01-29 15:46:12

I don't have anyhing in particular to discuss, but I'm happy to be there if anyone does. I added a note about the v0.1.22 release.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-01-29 15:51:11

If there's nothing else I think I'm going to give it a miss. I have another meeting after it and could do with a break today

👍 mboersma

👍:skin_tone_3: mcbenjemaa

Yongxiang Gao

2024-02-01 17:56:05

How to separate certain directory like /var into another partition and make this partition encrypted?
I try to add new partition for /var in file images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg

However, the output image file doesn't have the new partition.

Actually, in packer-plugin-qemu, it seems that, with ubuntu-2004 and ubuntu-2004-efi, the output image file is created from the same ISO image, but, the the output image file are really different (w/wo efi), how are the preseed files or other changes applied to it at all?

Drew Hudson-Viles

2024-02-02 08:34:45

Hi

It's passed in via userdata:

You should only need to make changes to the base/preseed-efi.cfg to make the required changes you desire.

The pressed is then passed in via the boot commands

Yongxiang Gao

2024-02-02 10:10:40

Thanks.
However, actually include
d-i preseed/include string ../base/preseed-efi.cfg

And I tried to add partitions like this in my local file packer/raw/linux/ubuntu/http/base/preseed-efi.cfg:

d-i partman-auto/expert_recipe string     <br>  gpt-boot-root ::              <br>   1 1 1 free               <br>     $bios_boot{ }            <br>     method{ biosgrub } .         <br>   200 200 200 fat32            <br>     $primary{ }             <br>     method{ efi } format{ } .      <br>   # 512 512 512 ext3            <br>   #  $primary{ } $bootable{ }       <br>   #  method{ format } format{ }      <br>   #  use_filesystem{ } filesystem{ ext3 } <br>   #  mountpoint{ /boot } .        <br>   5120 20000 -1 ext4           <br>     $primary{ } $bootable{ }      <br>     method{ format } format{ }      <br>     use_filesystem{ } filesystem{ ext4 } <br>     mountpoint{ / } .          <br>   1024 2048 4096 ext4         <br>     method{ format } format{ }      <br>     use_filesystem{ } filesystem{ ext4 } <br>     mountpoint{ /home } .        <br>   1024 2048 4096 ext4         <br>     method{ format } format{ }      <br>     use_filesystem{ } filesystem{ ext4 } <br>     mountpoint{ /var } .

The changes has no effect in the output image file: output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7, which always has 2 partitions:

Device                            Start    End  Sectors  Size Type
output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7p1   34 1050815 1050782 513.1M EFI System
output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7p2 1050816 16678878 15628063  5.5G Linux filesystem

Drew Hudson-Viles

2024-02-02 10:40:21

From what I can see, the preseed you have looks fine. I can't see any reason this would not working at a quick glance and I don't have time to test this locally at the moment but will when I get chance.

Can you connect via VNC (if that's an option during RAW builds) and see what's happening on each step? Can you see that step running?

Can you also confirm you've changed the line here to match the name of your recipe?

Yongxiang Gao

2024-02-02 11:33:09

Good catch.
However, after I change the name back to the original, the new partitions are still missing in the output image file.

Yongxiang Gao

2024-02-02 11:41:54

I will resort to VNC for sure.

👍 Drew Hudson-Viles

Pengfei Huang

2024-02-03 23:44:41

@Pengfei Huang has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 10:27:19

FYI: I'm creating a new image-builder release that includes this change to address a CVE 🙂 -

GitHub

Bump containerd to 1.7.13 by heytrav · Pull Request #1393 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1393

👍 Drew Hudson-Viles, mboersma

👍:skin_tone_3: mcbenjemaa

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 10:37:31

@Drew Hudson-Viles Do you want to see if you have the permissions to be able to /approve this PR? (I'm not sure if it's just maintainers or if reviewers are also able to)

GitHub

Image promotion for scl-image-builder v0.1.23 by AverageMarcus · Pull Request #6392 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.23 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers "@AverageMarcus @jsturtevant @kkes...

Original URL: https://github.com/kubernetes/k8s.io/pull/6392

Drew Hudson-Viles

2024-02-07 11:23:56

sure one tick

Drew Hudson-Viles

2024-02-07 11:24:22

oh Matt beat me to it 😄

😆 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 11:30:04

Documentation update -

GitHub

Update docs for release v0.1.23 by AverageMarcus · Pull Request #1394 · kubernetes-sigs/image-builder

What this PR does / why we need it: Updates the docs with reference to the v0.1.23 release. I also fixed the link to the testgrid job so it points at the correct one for building a tag image (previ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1394

👍 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 11:34:16

Can I get an /approve too please? 🙂

Drew Hudson-Viles

2024-02-07 11:34:24

I can try 😛

Drew Hudson-Viles

2024-02-07 11:34:51

DENIED 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 11:34:54

😭

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 11:35:43

I'm sure @mboersma will get to it when he's about 🙂 I'm about to head AFK but once that PR is merged could someone please announce it in the main channel? 🙏 The release is already created -

GitHub

Release v0.1.23 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.23">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.23</a> What's Changed Update docs after v0.1.22 release by @mboersm...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.23

👍 Drew Hudson-Viles

mboersma

2024-02-07 15:03:33

Thanks for pouncing on this @Marcus Noble and @Drew Hudson-Viles!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-07 17:49:57

I was just happy to see someone else had already done the PR with the CVE fix when I got there! 😉

Drew Hudson-Viles

2024-02-07 12:49:05

Image-builder v0.1.23 is now available:
Thanks to all contributors!

:thank_you: mboersma, Anurag

🎉 mboersma, Yongxiang Gao, Travis Holton

:partyk8s: mboersma

Yongxiang Gao

2024-02-08 20:27:29

Does image-builder support full disk encryption?

Drew Hudson-Viles

2024-02-08 20:32:22

It's not really image builder that would provide this but if it can be done via the preseed/cloud-init then yes it should support it.

Yongxiang Gao

2024-02-08 21:52:29

In case we are using the image (from image-builder) in tinkerbell action to create a EKS anywhere cluster on bare metal hosts.
What shall we do if we want to support encryption?

Yongxiang Gao

2024-02-08 22:01:04

The simplest way is to use the image (with preseed.cfg taken inside already), because we usually use stream-image action to install the OS.

Yongxiang Gao

2024-02-08 22:07:44

The problem is that, if I add the encryption part in the preseed-efi.cfg, I will get image-builder stuck as I reported on Jan 22 2024
I just file a ticket for such issue:

GitHub

With disk encryption added in preseed-efi.cfg, build stuck at waiting for SSH · Issue #1399 · kubernetes-sigs/image-builder

What steps did you take and what happened: [A clear and concise description on how to REPRODUCE the bug.] Add full disk encryption related hange to preseed-efi.cfg as follows: diff --git a/images/c...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1399

Drew Hudson-Viles

2024-02-08 22:46:04

Honestly, I'm not sure about this as I've not used preseed to setup encryption before.
When I get time I can take a look into it unless someone else comes up with something to help before then.
However, if it's hanging at waiting for SSH then it suggests an issue with the preseed itself which prevents it completing successfully. This means the VM will never reboot and launch using the generated disk image so that image-builder can proceed with the installation.

Yongxiang Gao

2024-02-08 23:52:11

Right, the VM reboot stucks

naadir

2024-02-09 16:07:20

can't help massively but i think your VM is prompting for a passphrase on the console and you'll need to automate that in Packer with the keyboard commands prior to it being able to SSH.

:ty: Yongxiang Gao

Ryan Gough

2024-02-10 17:56:46

@Ryan Gough has joined the channel

Ryan Gough

2024-02-10 17:58:34

Really enjoying the image-builder project, but i’ve started to modify the packer-node.json files for my use-case, which doesn’t feel correct. Is there a way to skip the export of OVA to OVF, and just template instead of the post-processors?

Ryan Gough

2024-02-10 18:02:05

I can see it sort of was addressed here a few years back

but nothing ever became of it, it seems.

GitHub

Add upload support for ova into vcenter · Issue #468 · kubernetes-sigs/image-builder

We are using the image builder internally. We have automated the upload process into our vcenter after building a new image. We are using govc: def setup_govc_env(vsphere_data): os.environ[ "GOVC_U...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/468#issuecomment-824901566

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-11 13:31:01

I’m not sure if this is what you’re asking exactly but rather than editing the existing files it’s possible to set the EXTRAPACKERVAR_FILES env var pointing to additional var files that layer on top of the built in ones.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-11 13:32:02

With regards to skipping the OVF. I’m not sure actually. I know when we build CAPV images we just ignore the other files and only copy what we need.

Ryan Gough

2024-02-11 13:34:21

Hi Marcus, i tried with copying packer-node.json, and using PACKERVARFILES but that didnt work, i’ll have a go with EXTRAPACKERVAR_FILESinstead?

Ryan Gough

2024-02-11 13:35:19

I only see two options really:

Ignore them, like you suggested,

Adjust the packer-node.json and remove the export definition in the clone block

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-11 13:47:52

I’m currently not at my laptop so can’t check but do you know if it’s possible to have packer not do the export? Is it something we could maybe have configured via env var or something? 🤔

Ryan Gough

2024-02-11 13:58:13

Doesn’t look like it; i mean the post-processing i thought about adding some env var logic on there, but the packer plugin doesn’t look like it’s able to disable the export option once set, i.e. enable true/false

😞 Marcus Noble

fad3t

2024-02-12 12:28:02

IIRC I had a look at something similar, and the current Packer config written in JSON comes with limitations

fad3t

2024-02-12 12:28:48

switching to HCL would allow to use dynamic blocks for example, to switch the export on/off in your case

fad3t

2024-02-12 12:28:52

but that's a lot of work 😕

fad3t

2024-02-12 12:29:51

I'm doing the same to add support for vApps, and for now I apply a patch to the packer-node.json file in the CI before calling the makefile. Dirty.. but it works 😛

Ryan Gough

2024-02-12 17:55:43

Yeah i think i’ll probably do this, at least it means maintaining a copy of the packer-node.json for now, but i see the only way this would work is if the vsphere packer plugin would support export.enable true/false

Lukas M

2024-02-11 13:23:16

@Lukas M has joined the channel

Slackbot

2024-02-11 13:24:11

This message was deleted.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-11 13:32:56

I don’t know much about kubevirt. Do you know what VM image type it needs? What makes you think that target isn’t enough?

Lukas M

2024-02-11 13:58:42

It should be QEMU, but with oem_id=kubevirt ( I guess ) to support and

Lukas M

2024-02-11 14:00:57

build-qemu-flatcar isnt enough, it does not support ignition, only coreos-cloudinit, that's not such a problem, but coreos-cloudinit does not handle # jinaja template string on the first line of userData generated by image-builder machinery

Lukas M

2024-02-11 14:04:37

I`ve tried qemu-flatcar target with OEM_ID=kubevirt, ..its there

test /home/core # cat /usr/share/oem/grub.cfg
set oem_id="kubevirt"

, but still not working..

Lukas M

2024-02-11 14:06:59

Hm, its probably not enough, , also flatcar.oem.id has to be set, not sure where exactly

Lukas M

2024-02-11 14:09:26

Its there also:

test /home/core # cat /proc/cmdline
rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 flatcar.first_boot=detected flatcar.oem.id=kubevirt verity.usrhash=d8aba28f890e180820484397bf8fd4ea722445662d25e7a2139360f12f74fa58

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-11 14:13:28

So the OEM is correctly being set but the images still aren’t working for you? What’s no working exactly? An error or just not able to boot at all?

Drew Hudson-Viles

2024-02-12 11:25:03

I haven't done the KubeVirt side of things with image builder just yet but plan on doing so this or next week for Ubuntu, so can't help too much on the Flatcar side but there is a kubevirt script in the qemu/scripts directory that may be of use?

Lukas M

2024-02-12 11:43:51

Its working with build-qemu-flatcar target, sorry for the fuzz...

Drew Hudson-Viles

2024-02-12 11:44:18

No worries!

Slackbot

2024-02-12 15:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-12 15:32:44

☝️ Agenda is currently empty. Is there anything anyone would like to discuss or should we skip?

mboersma

2024-02-12 15:58:42

I was thinking we should discuss EKS-Anywhere, but IDK what else there is to say once I thought about it: we just have to "return to sender" unless it's a bug reeproducible in image-builder on its own.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-12 15:59:14

Yeah lets do that!

👍 mboersma

mboersma

2024-02-12 15:59:45

Also I'm reworking the Azure pipelines to just be GH Actions and was wondering if that crossed a line or not. I'll put that on the agenda, should be a short discussion.

👍 Drew Hudson-Viles, Marcus Noble

Ryan Gough

2024-02-12 17:54:40

man i missed it!

mboersma

2024-02-12 19:59:16

There's a recording here:

YouTube

ogghead (https://www.youtube.com/@ogghead)

SIG Cluster Lifecycle - Image Builder Office Hours - 2024-02-12

Original URL: https://youtu.be/y-QuXkFmeD8

✅ Ryan Gough

Kazuya Nomura

2024-02-12 21:12:18

@Kazuya Nomura has joined the channel

Yongxiang Gao

2024-02-14 15:39:16

I see magic output image of ubuntu efi 2004 with partitions which doesn't honor preseed-efi.cfg

I used DEBUG=1 PACKER_LOG=1 to narrow down the partitions are generated at this step:

==> qemu: Pausing after run of step 'stepTypeBootCommand'. Press enter to continue. 
  qemu: Not using a NetBridge -- skipping StepWaitGuestAddress

Do you know what else can we try (e.g. increase verbose level, etc) to narrow down further?

mboersma

2024-02-14 19:40:56

You can try adding PACKERFLAGS=-debug. Anything in $PACKERFLAGS gets passed to the packer build command, and -debug is suggested by .

developer.hashicorp.com

Debugging - Other | Packer | HashiCorp Developer

Packer strives to be stable and bug-free, but issues inevitably arise where certain things may not work entirely correctly, or may not appear to work correctly.

Original URL: https://developer.hashicorp.com/packer/docs/debugging

:ty: Yongxiang Gao

Makesh k

2024-02-16 09:27:34

@Makesh k has joined the channel

Ricky Sadowski

2024-02-20 14:02:19

@Ricky Sadowski has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-20 21:49:15

I’d like to get a new release put out as we’ve had a handful of fixes come in since the last. Is there any active PRs anyone would like to try to get in before I do?
I’m hoping to do a new release tomorrow if there’s no objections?

Drew Hudson-Viles

2024-02-20 21:50:16

Nothing from my side

mboersma

2024-02-20 22:54:13

Fine with me! There is little downside to doing frequent releases IMHO.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-20 22:58:13

Yeah. Just wanted to see if any PRs were ready for review before I do. 🙂
I mainly want to get the python fix for Azure Flatcar images out as we’re currently blocked by that at GS 😜

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-21 11:49:18

I'm kicking this off now. 🚀

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-21 12:03:04

Promo PR -

GitHub

Image promotion for scl-image-builder v0.1.24 by AverageMarcus · Pull Request #6449 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.24 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers "@AverageMarcus @jsturtevant @kkes...

Original URL: https://github.com/kubernetes/k8s.io/pull/6449

Drew Hudson-Viles

2024-02-21 12:25:57

aaah I can't approve that one - thought I'd try for ya!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-21 12:26:20

Yeah, maintainers only 😞 Thanks for trying though 😄 I can wait for the others.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-21 12:36:47

Docs update for when the release is published (also include a shiny new script to automate most of this PR in the future 😉)

GitHub

Update docs for 0.1.24 & include new script to handle the docs updates by AverageMarcus · Pull Request #1411 · kubernetes-sigs/image-builder

Change description Updated docs with reference to the new v0.1.24 release Introduced a new script (and make target) to make this PR easier in the future. Related issues N/A Additional context N/A

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1411

🙌 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-21 14:50:14

Thanks y'all! 🙂

I'm still annoyed by the amount of manual steps needed to do a release. I really wish we could have the whole thing automated just from pushing a new tag. 🤔 Not sure how we'd handle the "wait for promo PR" and "wait for image being pullable" though.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-21 14:45:16

Image-builder v0.1.24 is now available:

Thanks to all contributors! 🎉 💙

GitHub

Release v0.1.24 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.24">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.24</a> What's Changed Update docs for release v0.1.23 by @AverageMa...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.24

🎉 mboersma, Travis Holton

:thank_you: mboersma

Shashinandan Srinivasa

2024-03-06 18:05:26

Hi, sorry if my query looks novice. In the release.md file for v.0.1.24, why does it say the "current release of Image builder is v.0.1.23"?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-06 18:08:44

The release note and docs are updated after the release is available.

👍 Shashinandan Srinivasa

Hrvoje Bašić

2024-02-23 14:59:03

@Hrvoje Bašić has joined the channel

Slackbot

2024-02-26 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-26 16:31:01

☝️ Starting now 🙂 We've got a few items on the agenda for today.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-26 16:57:05

Huddle in thread about eks-anywhere 🧵

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-26 16:57:16

A huddle started

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-26 16:58:12

@Abhay Krishna Arunachalam

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-26 17:42:16

FYI - The general outcome with regards to image-builder was:
Regarding incoming issues - we’ll send people to the EKS-anywhere project (as they also want to be aware of issues with their CLI) and they will triage and open issues with us as and when needed. If we find this not manageable we can then look into having some of the EKS-anywhere members as a group within image-builder that we can assign issues to.
(cc @mboersma @jsturtevant @kiran keshavamurthy)

🙌 Abhay Krishna Arunachalam

:ty: jsturtevant

👍 kiran keshavamurthy, mboersma

orin

2024-02-26 17:11:16

@orin has joined the channel

Nicolò Ciraci

2024-02-27 10:51:25

@Nicolò Ciraci has joined the channel

Nicolò Ciraci

2024-02-27 10:52:52

Hello! I would like to start saying that I’m already sorry if this is not the correct way/channel to ask, but are other kinds of CRI runtime “supported”? I would like to build an image with CRI-O and PR the project. Thanks a lot!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-27 11:01:07

Only containerd is currently supported as installing it is currently one of the main tasks performed.
I suspect it might be possible to install CRI-O after image-builder has finished and switch to using that by default instead but as far as I know there's no way to configure image-builder to install a different CRI.

Nicolò Ciraci

2024-02-27 11:09:35

I was thinking about editing the ansible playbook to implement a ‘if-then’ logic to support other CRI flavour - this would be like implementing more optional variables and running the correct playbooks. Is this offlimits?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-27 11:16:08

Not at all. We'd welcome such a contribution. Just be aware that I think it would mean a lot of the vars files would need updating for all the providers to be able to support passing in the needed versions, sha's etc.

Nicolò Ciraci

2024-02-27 12:24:01

Ok, then I will work on this implementation and share the results. Hoping to be able to implement something quickly!

👍 Marcus Noble, jsturtevant

🤞 Marcus Noble

jsturtevant

2024-02-27 17:23:13

This should be possible to cofgirgure multi runtimes In a fairly clean way with ansible. on windows side we initially had two different runtimes, it's been removed now but was not to bad when it was done.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-02-27 17:51:35

Oh interesting. I wasn't aware of that. Might be worth taking a look through the git history to see how it was handled previously then.

Nicolò Ciraci

2024-02-27 17:57:12

Oh interesting , I’ve already implemented something and I will let you know tomorrow if it works as intended 😅 Should I refer to the ansible playbook for the older implementation?

jsturtevant

2024-02-27 18:52:04

Np, we can go with what you have and tweak from there

👍 Nicolò Ciraci, Marcus Noble

Nicolò Ciraci

2024-02-29 09:48:16

Hi! Right now I’m able to build a qemu backed image that leverage crio, I will implement gvisor for equal compatibility as containerd and then open a PR to get a review. If I can manage I will try also to prepare multiple version of builds to get a better coverage of os

👍 jsturtevant

Yongxiang Gao

2024-02-29 19:29:45

How to add a custom file to the image with image-builder code?

Abhay Krishna Arunachalam

2024-02-29 19:34:06

If this is related to using image-builder with EKS Anywhere, you can refer to this doc.

EKS Anywhere

Artifacts

Artifacts associated with this release: OVAs and images.

Original URL: https://anywhere.eks.amazonaws.com/docs/osmgmt/artifacts/#mounting-additional-files

:ty: Yongxiang Gao

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:20:57

🤔 If we don't have that in base image-builder I think it would be a nice thing to add.

Abhay Krishna Arunachalam

2024-03-01 08:22:52

yeah I was thinking about that too. We use an additional role in the EKS-A repo for copying files.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:25:03

We have a similar one for loading additional executables from remote sources () but I think it would be nice to also have the same role you have.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:31:04

/cc @Drew Hudson-Viles you might be interested in this ☝️

Drew Hudson-Viles

2024-03-01 08:36:33

I'm not aware of any approach we have at the moment. It's certainly something we can look into though!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:43:54

I suspect we can likely copy over that role without much trouble.

👍 Drew Hudson-Viles

Yongxiang Gao

2024-03-03 18:07:10

Actually, I find a way to add with following:

ansible/roles/sysprep/files

ansible/roles/sysprep/tasks/

Drew Hudson-Viles

2024-03-04 09:25:15

That will work in a fork such as yours however it's not something that's easily adaptable for people using the core code so we will look into this.

Drew Hudson-Viles

2024-03-04 09:27:59

I've raised this here anyway and will look into this as soon as I can

GitHub

Add custom files into the image · Issue #1422 · kubernetes-sigs/image-builder

Is your feature request related to a problem? Please describe. A user has requested that we support adding custom files into the image builder code. the EKS Anywhere supports this already so we sho...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1422

:thx_thanks: Marcus Noble

Abhay Krishna Arunachalam

2024-03-01 08:08:34

I noticed that a handful of ISO URLs in the Packer config files were returning 404s because of the images being removed from the mirror/release endpoint. I opened a PR to fix that and also update to latest point releases for some others. I was also thinking of switching the ubuntu 20.04 ISO URLs from the cdimage.ubuntu.com to the old-releases.ubuntu.com domain for consistency with ubuntu 22.04, any concerns with that?

GitHub

Fix ISO URLs pointing to missing images by abhay-krishna · Pull Request #1420 · kubernetes-sigs/image-builder

Updating a few ISO URLs pointing to images that got removed from the downloads mirror.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1420

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:22:01

This has been an ongoing pain 😞
Ubuntu doesn't include (or at least didn't used to) the latest release at the old-releases.ubuntu.com endpoint. So if we wanted to the latest release we needed to use the one that ended up breaking.

Abhay Krishna Arunachalam

2024-03-01 08:25:06

Yeah I have been thinking about how to solve this one too. But in this PR, I'm updating to the latest one available in old-releases, that should be okay right?

Abhay Krishna Arunachalam

2024-03-01 08:26:11

i'll update the checksums, thanks

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:26:15

Ah nice! I did have an issue for it 😆
Yeah, bumping the versions is always welcome, just remember that the checksums also need updating.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:26:20

ha, yeah

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:26:36

We do have an issue where we'd like to automate this but it's the checksums thats currently blocking that.

GitHub

Automated dependency updates · Issue #1283 · kubernetes-sigs/image-builder

We've had several PRs lately to bump versions of various dependencies that we've noticed have become several versions out of date. We tend to do this on an ad-hoc basis when we remember to or someo...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1283

Abhay Krishna Arunachalam

2024-03-01 08:26:45

i was wondering if we should have a periodic for updating ISO URLs

Abhay Krishna Arunachalam

2024-03-01 08:28:02

we could compute the checksums on the fly right? although it would take some time given the size of the images

Abhay Krishna Arunachalam

2024-03-01 08:28:35

or maybe parse the SHA256 file in the same releases endpoint

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:29:00

Hmm... that's not a bad idea. We might not actually have to calculate them... yeah, exactly that 😆 I feel like we're thinking the same

🙌 Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-03-01 08:39:13

Is there any reason not to switch from the ubuntu-legacy-server (available at cdimage.ubuntu.com) to the ubuntu-live-server (Available at old-releases.ubuntu.com) ISOs for ubuntu 20.04? I was thinking it'll be nice to standardize the ubuntu releases endpoints and not have separate sources?

Abhay Krishna Arunachalam

2024-03-01 08:43:09

also I think we can standardize URLs like these

and

I confirmed both are the same thing by checking the sha256 sum and the Entity tag (Etag) returned by curl , so we can use just one

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 08:43:38

I don't think so. I suspect its just a case of different people working on different areas leading to inconsistency

Abhay Krishna Arunachalam

2024-03-01 08:55:40

Makes sense, I will try to include that change in my PR

Abhay Krishna Arunachalam

2024-03-01 08:56:57

Also I think depending on the old-releases images isn't a bad thing, since they will get upgraded to the latest point release when image-builder runs the dist-upgrade step

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-01 09:10:00

True, didn't think about that.

Abhay Krishna Arunachalam

2024-03-02 00:27:01

Opened this PR to fix/update some ISO URLs and add a script for updating checksums in the future. Would appreciate some feedback on this. Thanks!

GitHub

Update ISO URLs to latest, add script to update Ubuntu ISO checksums by abhay-krishna · Pull Request #1420 · kubernetes-sigs/image-builder

Updating a few ISO URLs pointing to images that got removed from the downloads mirror Bump some Ubuntu ISO URLs to the latest stable point releases Add script to update Ubuntu ISO URL checksums

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1420

Abhay Krishna Arunachalam

2024-03-02 00:35:28

Packer allows us to specify the iso_checksum as a URL pointing to a checksums file containing the actual checksum. I propose making that switch instead of hardcoding the checksum, to avoid having to update them each time. I saw we do it for Flatcar Linux, but we should ideally extend it for other OSs too.

I haven't made the proposed change in the PR, but I can update it if it sounds like a reasonable change.

👍 mboersma

RAKESH BOINAPALLY

2024-03-04 18:31:11

Quick question i'm running into following error did any one face this error in the past any lead

Build 'vsphere-iso.vsphere' errored after 17 minutes 57 seconds: error exporting vm: ServerFaultCode: Permission to perform this operation was denied.

Abhay Krishna Arunachalam

2024-03-04 18:52:55

error exporting vm

so is this occuring at the end of the image build?

Could you verify that the vsphere user has these privileges?

RAKESH BOINAPALLY

2024-03-04 19:12:03

End of the image

RAKESH BOINAPALLY

2024-03-04 19:33:06

it is failing at this particular step ==> vsphere-iso.vsphere: Goss validate ran successfully
==> vsphere-iso.vsphere:
==> vsphere-iso.vsphere:
==> vsphere-iso.vsphere:
==> vsphere-iso.vsphere: Downloading spec file and debug info
vsphere-iso.vsphere: Downloading Goss specs from, /tmp/goss-spec.yaml and /tmp/debug-goss-spec.yaml to current dir
==> vsphere-iso.vsphere: Executing shutdown command...
==> vsphere-iso.vsphere: Deleting Floppy drives...
==> vsphere-iso.vsphere: Deleting Floppy image...
==> vsphere-iso.vsphere: Eject CD-ROM drives...
vsphere-iso.vsphere: Starting export...
==> vsphere-iso.vsphere: Provisioning step had errors: Running the cleanup provisioner, if present...
==> vsphere-iso.vsphere: Clear boot order...
==> vsphere-iso.vsphere: Power off VM...
==> vsphere-iso.vsphere: Destroying VM...
Build 'vsphere-iso.vsphere' errored after 17 minutes 33 seconds: error exporting vm: ServerFaultCode: Permission to perform this operation was denied.

==> Wait completed after 17 minutes 33 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso.vsphere: error exporting vm: ServerFaultCode: Permission to perform this operation was denied.

RAKESH BOINAPALLY

2024-03-05 21:02:27

it was a permission issue things did go fine

RAKESH BOINAPALLY

2024-03-05 21:02:33

after fixing the permission issue

Abhay Krishna Arunachalam

2024-03-05 22:26:45

gr-nice-2

snevedomski

2024-03-05 16:14:53

@snevedomski has joined the channel

snevedomski

2024-03-05 16:52:56

Hi all. Need your help with proxmox image-builder.
Followed this instruction:
But I'm getting "write tcp 192.168.30.33:59042->192.168.30.2:8006: write: broken pipe" error.

And I don't even get how it is supposed to work. Instruction doesn't provide password or token secret. Tried adding them as env vars. Nothing changes.
What am I doing wrong?

image.png

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-05 17:06:08

@mcbenjemaa are you able to help here?

👍 snevedomski

👍:skin_tone_3: mcbenjemaa

mcbenjemaa

2024-03-05 19:02:57

As i can see, this is network issues happens in your setup.

You will need to rerun or you can actually change the values so you can use an existing ISO.

snevedomski

2024-03-05 20:06:01

Thank you. I'll try it.

snevedomski

2024-03-05 22:31:05

Rerun didn't help
But I figured out my problem.
PROXMOXUSERNAME is proxmox token ID
PROXMOXTOKEN is proxmox token secret
This is very counterintuitive.

Thanks everyone for help

👍:skin_tone_3: mcbenjemaa

Abhay Krishna Arunachalam

2024-03-05 22:47:07

You can refer to the Packer Proxmox builder docs for the syntax and semantics of the username and token fields

developer.hashicorp.com

Proxmox Builder | Integrations | Packer | HashiCorp Developer

Explore Packer product documentation, tutorials, and examples.

Original URL: https://developer.hashicorp.com/packer/integrations/hashicorp/proxmox/latest/components/builder/iso#optional:

👍 snevedomski

mcbenjemaa

2024-03-07 11:58:58

Oh, yeah exactly.
Packer plugin uses Username/password and token
But in capi provider we only use Token based authentication.

RAKESH BOINAPALLY

2024-03-06 18:15:58

Hi All did any one use govc to export ova from one vcenter locally then import it to different vcenters.It would be very helpful if some has done this kind of stuff so that dont need to reinvent the wheel.I'm trying to use govc to export and import ova created using image builder

Shashinandan Srinivasa

2024-03-06 21:37:51

gr-wave_animated the last image-builder version that works in my CI to build custom vsphere images is v0.1.19. Even the latest one (v0.1.24) just hangs at "Waiting for ssh to be available". Any pointers to troubleshoot this behavior?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-06 21:45:29

If 1.19 is the last that worked for you then I suspect something in this release broke for you: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.20

I see a couple vsphere changes there that might give you some insight into what might be wrong.

It also might be useful if you share what make target you’re using, what vars you’re providing and if you’re using the container image to run image-builder or not.

I can confirm that, at least for me, the latest release was able to successfully build a flatcar image for vsphere.

GitHub

Release v0.1.20 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.20

Shashinandan Srinivasa

2024-03-06 22:02:47

"# make build-node-ova-vsphere-ubuntu-2204" Its a gitlab pipeline with following steps

Shashinandan Srinivasa

2024-03-06 22:02:54

I clone the tag, in this case v0.1.19 -> add custom ansible roles -> build docker image -> run the make target using the new built docker image

Shashinandan Srinivasa

2024-03-06 22:05:00

I will try using the docker image that you have directly and get back. Its flaky for sure. I have tried running this directly on host, without any container. Sometime it works and sometimes it doesn't

Shashinandan Srinivasa

2024-03-06 22:48:37

From the VM console, this is where it stops and doesn't proceed. On the cli its waiting for ssh

Shashinandan Srinivasa

2024-03-06 22:48:41

*Thread Reply:* None

image.png

Shashinandan Srinivasa

2024-03-06 23:16:19

So, direct checkout to host and running it works.

Shashinandan Srinivasa

2024-03-06 23:17:32

If I try to create a custom docker image and run make target from a container out of it, thats when it hangs waiting for ssh. Any setting I am missing on the docker host or docker build that will help here?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-07 06:51:13

Are you building your docker image on top of ours and then just adding in the ansible roles you need? Or are you building from scratch?

Shashinandan Srinivasa

2024-03-11 18:12:13

Building from scratch and I have used your Dockerfile as reference. Only difference is that I clone specific tag. "git clone --depth 1 --tags v0.1.24 --single-branch"

Shashinandan Srinivasa

2024-03-11 18:13:47

I removed the above step of cloning just a single tag and cloned entire repo. The build happens without issues. It takes some time, but it does build successfully.

Shashinandan Srinivasa

2024-03-11 18:14:24

I will update when I find more details about this.

Mitchel Haring

2024-03-07 21:57:31

Any thoughts on adding Rocky 9 Linux for OVA? -

GitHub

Rocky LInux 9 OVA by 0nebody · Pull Request #1387 · kubernetes-sigs/image-builder

What this PR does / why we need it: Adds Rocky Linux 9 for vSphere OVA. Additional context Defaulted firmware to EFI for users with large GPUs or multiple vGPUs.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1387

👍 mboersma

Yongxiang Gao

2024-03-10 08:31:28

How to override kubernetes version in the output image, I know with image-builder command, we can use option "release-channel" as follows:

image-builder build --os ubuntu --os-version 20.04 --hypervisor baremetal --release-channel 1-28 --firmware efi

How about the command "make build-raw-ubuntu-2004-efi" from the source of ?

Yongxiang Gao

2024-03-10 08:36:33

With this prefix?

PACKER_FLAGS="--var 'kubernetes_rpm_version=1.28.3' --var 'kubernetes_semver=v1.28.3' --var 'kubernetes_series=v1.28' --var 'kubernetes_deb_version=1.28.3-1.1'"

Drew Hudson-Viles

2024-03-10 08:37:09

Hi,

If you look in the packer.json file, you'll be able to see a bunch of variables you can override.

For example the Kubernetes version is overridden using this one: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/raw/packer.json#L164.

In your variables file, you can add it and it will override it.
https://github.com/kubernetes-sigs/image-builder/blob/main/docs/book/src/capi/capi.md?plain=1#L97

You can also override them directly with flags.
https://github.com/kubernetes-sigs/image-builder/blob/main/docs/book/src/capi/capi.md?plain=1#L91

:ty: Yongxiang Gao

mboersma

2024-03-11 14:51:33

Reminder: Image-Builder office hours start in 40 minutes. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:55:03

Oh... we're in the fun two weeks of DST mis-match 😆

mboersma

2024-03-11 14:56:22

Oh...yikes. Should we go with Pacific time or wait an hour?

mboersma

2024-03-11 14:56:54

I wondered why I didn't see Slackbot, maybe we should wait an hour?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:56:55

My calendar entry is correct (in 40 min) but it's an hour earlier than it normally is for me.
The time still works for me though.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:57:17

Although the doc does say UTC thinking

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:57:32

Oh wait, that's right

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:57:45

3:30pm UTC is in ~30 min

mboersma

2024-03-11 14:58:17

I am very much a morning person, but the Monday when DST changes just sucks. 🙂 Ok, see you in 30 minutes.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:58:35

I just suck at timezones in general 😆

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 14:58:55

Looks like when I set up the slack reminder it was during DST

👍 mboersma

Slackbot

2024-03-11 15:30:15

Reminder: Image-Builder office hours start in 1 hour. Agenda:

😀 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-11 15:30:51

☝️ Ignore, it's actually now

giridhar M Sharma

2024-03-11 22:01:28

@giridhar M Sharma has joined the channel

giridhar M Sharma

2024-03-11 22:08:43

Hello All,
we are trying to build a ubuntu 22.04 on vsphere 8.
I attended the office hours meeting today and discussed this problem. There were few threads which were suggested. I tried everything but still it doesn't work.
Threads I went through

The first one seems somewhat different problem. Second thread matches the problem I am facing. There was a suggestion to make changes to the bootcommandprefix in one of the workarounds. With this at-least the ssh is becoming available (manually), however, make still getting stuck at Waiting for SSH to become available... . If I go and look at the console, it would be stuck in choose the language screen.

I am just thinking, is the boot command for ubuntu22.04 correct or something is missing here.
boot command for 22.04 given in packer/ova/ubuntu-2204.json is,

"boot_command_prefix": "clinux /casper/vmlinuz ipv6.disable={{ user boot&#95;disable&#95;ipv6 }} --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/22.04/'initrd /casper/initrdboot",

#1076 build-qemu-ubuntu-2204 stuck in "Waiting for SSH to become available..."
kubernetes-sigs/image-builder | Feb 15th, 2023 | Added by GitHub

#1301 [ova] ubuntu22 won't build on vsphere 7.x
kubernetes-sigs/image-builder | Sep 20th, 2023 | Added by GitHub

GitHub

build-qemu-ubuntu-2204 stuck in "Waiting for SSH to become available..." · Issue #1076 · kubernetes-sigs/image-builder

Hi, I installed the image builder based on this doc. Then to building an image for Openstack I use this doc. But with this command make build-qemu-ubuntu-2204 it stucks in the ssh step. My operatin...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1076

GitHub

[ova] ubuntu22 won't build on vsphere 7.x · Issue #1301 · kubernetes-sigs/image-builder

What steps did you take and what happened: [A clear and concise description on how to REPRODUCE the bug.] Clone the image-builder repository with tag 0.1.19 Go to the folder image-builder/images/ca...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1301

GitHub

GitHub - kubernetes-sigs/image-builder: Tools for building Kubernetes disk images

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder

snevedomski

2024-03-11 22:41:24

That's what I use.
"clinux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/22.04/'initrd /casper/initrd boot "
Works well for me.

snevedomski

2024-03-11 22:46:36

Your boot command is probably fine too.
You get language screen after some time if you cannot establish SSH connection or if you boot from install disk after reboot.
Check SSH related variables and boot order

giridhar M Sharma

2024-03-11 22:49:50

Did you happen to do any edits in the image builder configs to make this work? The same vpshere is building 2004 ova just fine.

snevedomski

2024-03-11 23:02:31

nope, But I'm using proxmox image-builder

giridhar M Sharma

2024-03-11 23:12:26

ah ok. So this issue is something specific to vsphere then

giridhar M Sharma

2024-03-12 19:06:58

Hi @snevedomski: Till I get some solution on the vsphere, I thought I will give it a shot with proxmox as it is working for you as is. I created a VM out of the published ISO. However when I run the build command, I hit this issue and I have no idea why its failing. Any hints here will be of great help

==> proxmox-iso.ubuntu-2204: Post "https://:8006/api2/json/nodes/pve/storage/local/upload": write tcp :58174->proxmox_vm_ip:8006: write: broken pipe
Build 'proxmox-iso.ubuntu-2204' errored after 4 seconds 920 milliseconds: Post "https://:8006/api2/json/nodes/pve/storage/local/upload": write tcp :58174->10.206.140.162:8006: write: broken pipe

snevedomski

2024-03-13 01:03:07

try to connect to proxmox server from terminal.

Justinas B

2024-07-08 10:04:09

Hey @snevedomski , is it still working for you?

Justinas B

2024-07-08 10:04:32

my build for proxmox also gets stuck on language selection screen and i cannot find a way around it

Ahree Hong

2024-03-13 17:41:45

@Ahree Hong has joined the channel

Ahree Hong

2024-03-13 18:14:03

Hello all!
I opened a PR to upstream a patch we added in eks-anywhere which adds qemu support for RHEL9

GitHub

Add QEMU support for RHEL 9 by ahreehong · Pull Request #1425 · kubernetes-sigs/image-builder

Change description This PR adds RHEL 9 support for QEMU.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1425

👍 mboersma

:thank_you_icon: Marcus Noble

Abhay Krishna Arunachalam

2024-03-13 18:16:47

Hiya @Ahree Hong!gr-wave_animated

Drew Hudson-Viles

2024-03-13 20:55:29

Image-builder v0.1.25 is now available:
Thanks to all contributors! 🎉

GitHub

Release v0.1.25 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.25">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.25</a> What's Changed Update docs for 0.1.24 & include new script t...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.25

:parrotk8s: Marcus Noble, Vignesh Goutham, Mitchel Haring, Abhay Krishna Arunachalam, jsturtevant

Vignesh Goutham

2024-03-13 21:35:38

Hello everyone, inspired from @Ahree Hong’s contribution, a PR to support Raw RHEL-9 both bios and efi. Please take a look.

GitHub

Add raw image building support for rhel-9 and rhel-9 efi by vignesh-goutham · Pull Request #1427 · kubernetes-sigs/image-builder

Change description Adding Image building support on Raw provider for RHEL-9 and RHEL-9 EFI. As part of the change, the kickstart files for RHEL raw are now symlinked to qemu's RHEL kickstart files...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1427

:thank_you_icon: Marcus Noble

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:30:50

😞 We really need to come up with a good solution for the ubuntu ISO URLs.
Another instance of confusion regarding using the old url instead of latest -

Anyone have any ideas on how we can have the URL static but still use the latest available? I'm surprised Ubuntu doesn't provide this.

Abhay Krishna Arunachalam

2024-03-14 08:47:10

Really not loving this hacky approach but it does give the latest available one on the releases endpoint

$ curl -L  | grep -o 'href="ubuntu-22.04.-live-server-amd64.iso">' | gsed -e "s/href=\"//g" | gsed -e 's/">//g' | uniq
ubuntu-22.04.4-live-server-amd64.iso

$ curl -L  | grep -o 'href="ubuntu-20.04.-live-server-amd64.iso">' | gsed -e "s/href=\"//g" | gsed -e 's/">//g' | uniq
ubuntu-20.04.6-live-server-amd64.iso

we could get the value dynamically and jq it into the packer config file before the build

Abhay Krishna Arunachalam

2024-03-14 08:48:25

Keep in mind, this only addresses the issue of URL availability, not build reproducibilty.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:48:39

But that URL will eventually stop working which would mean image-builder breaks without changes 😞

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:48:49

Ha, yeah exactly 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:50:03

What I really want is for Canonical to provide an endpoint that we can give a version to and it redirects to the appropriate URL.

✅ Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-03-14 08:51:06

But that URL will eventually stop working

you mean the ISO URL will return 404? or it will point to a bad ISO?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:51:36

Return 404 once a new update is available.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:51:50

The releases.ubuntu.com only makes the latest available

Abhay Krishna Arunachalam

2024-03-14 08:52:46

The above curl command bypasses that by getting the directory listing at the time of build (rather just before), and setting the ISO URL to that.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:53:08

Oh you mean at time of running the make targets

Abhay Krishna Arunachalam

2024-03-14 08:53:20

we're querying so if it gets updated to 22.04.5, it would still get that

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:53:55

But then we have the problem that the same version of image-builder could introduce new changes because a new Ubuntu version was release. 😞

Abhay Krishna Arunachalam

2024-03-14 08:54:07

in that case, we don't have the issue of 404, but image-builder would still break if the new ISO is broken

Abhay Krishna Arunachalam

2024-03-14 08:54:11

yup

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 08:55:37

I might see if I can find a contact at Canonical to discuss it with them 🤔

🙌 Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-03-14 09:01:10

we could still pin to the latest releases version (possibly in a Makefile or tag files like we use in EKS Anywhere) but have some automation around it that checks for the latest releases periodically and updates the pinned version. Then atleast we have the auditability.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 09:03:07

Looks like Canonical have a booth at KubeCon next week so I might try and see if I can find someone appropriate to talk with.
Failing that, I might see if we could get a proxy service hosted in the Kubernetes community cluster to handle this.

Abhay Krishna Arunachalam

2024-03-14 09:04:42

Nice! I was looking into other mirrors too, but no one seems to have the combination of old and new releases images (why would they, I guess)

mboersma

2024-03-14 15:40:37

Failing that, I might see if we could get a proxy service hosted in the Kubernetes community cluster to handle this.

That was the only solution I could think of here--maintain our own redirect URLs. It would put a maintenance burden on the image-builder team, but at least we wouldn't have to update code every time it changes.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-14 15:41:49

If it's something we can run on the community cluster it should be a fairly simple application. But I'm much prefer Canonical actually did it rather than us if possible

👍 mboersma, Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-03-18 17:02:18

Hi all, looking for an approver to approve this.

GitHub

Revert updates to Ubuntu 2004 ISO URLs by abhay-krishna · Pull Request #1431 · kubernetes-sigs/image-builder

This PR reverts updates to Ubuntu 20.04 URLs changed in #1420 since we need to figure out a way to migrate these to the subiquity installer from the debian-installer preseed approach before the URL...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1431

Abhay Krishna Arunachalam

2024-03-18 18:36:12

Hello all, I opened this PR to move the ks configs for RHEL raw builds inside the http directory since that's what Packer expects when serving these to the VM

GitHub

Place RHEL kickstart files for raw builds in `http` directory by abhay-krishna · Pull Request #1432 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1432

👍 mboersma

Abhay Krishna Arunachalam

2024-03-19 16:17:02

cc @mboersma @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-19 16:20:17

Sorry, I saw your PRs but I’m away at Kubecon this week and haven’t had a chance to take a look yet. @Drew Hudson-Viles maybe able to take a look?

Abhay Krishna Arunachalam

2024-03-19 16:38:31

No worries, apologies for the noise. Wanted to get these two merged because they could be potential blockers for image builder users

:thank_you_icon: Marcus Noble

Drew Hudson-Viles

2024-03-19 16:39:42

I'll be able to take a look shortly, no problem

:ty: Abhay Krishna Arunachalam

galop

2024-03-19 08:42:25

@galop has joined the channel

rodrigodelmonte

2024-03-19 22:53:32

@rodrigodelmonte has left the channel

Abhay Krishna Arunachalam

2024-03-19 23:25:33

Opened a cleanup PR for symlinks and unused kickstart files

GitHub

Replace symlink OS folders with actual folders, remove unused kickstart/preseed configs by abhay-krishna · Pull Request #1433 · kubernetes-sigs/image-builder

This PR extends some of the work done in #1427 to replace the OS folder symlinks with the actual folders and content to isolate changes to a specific OS or OS version and avoid breaking something u...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1433

RAKESH BOINAPALLY

2024-03-21 16:27:33

i was running image-builder with replacing old-ubuntu-releases with seems like at vm level in vsphere it is asking to make choice english and presss enter did any one ran into such issue

releases.ubuntu.com

Ubuntu 22.04.4 LTS (Jammy Jellyfish)

CD images for Ubuntu 22.04.4 LTS (Jammy Jellyfish)

Original URL: https://releases.ubuntu.com/jammy/

Abhay Krishna Arunachalam

2024-03-21 17:16:12

If it's prompting you then it means the Packer VM did not get the autoinstall configuration and is dropping into the interactive Ubuntu install, which is not desired.

RAKESH BOINAPALLY

2024-03-21 17:56:05

that is strange all i was doing is passing ubuntu iso check sum and url as environment variables

RAKESH BOINAPALLY

2024-03-21 20:29:09

Also im noticing Buffer I/O error on disk dev fd0

RAKESH BOINAPALLY

2024-03-21 20:29:27

which seems to be using floppy disk in vsphere any option to avoid floppy disk

RAKESH BOINAPALLY

2024-03-22 00:51:56

Seems like this is the issue that i'm running into

GitHub

[ova] ubuntu22 won't build on vsphere 7.x · Issue #1301 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1301

RAKESH BOINAPALLY

2024-03-22 18:46:06

any ideas folks

RAKESH BOINAPALLY

2024-03-22 18:46:22

i tried cd_files option still it is asking for english language selection

Abhay Krishna Arunachalam

2024-03-22 19:47:51

@mboersma could you give me an /approve here?

GitHub

Revert updates to Ubuntu 2004 ISO URLs by abhay-krishna · Pull Request #1431 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1431

👍 mboersma

Slackbot

2024-03-25 15:30:26

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2024-03-25 15:32:18

I'm afraid I'll be unable to make this one due to a house viewing I need to attend.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-25 15:44:07

The agenda is currently empty so I suggest we skip this one then. (I'm also not feeling too great so happy to go rest instead 😛 )

👍 Drew Hudson-Viles, mboersma

Drew Hudson-Viles

2024-03-25 16:00:57

Works for me. Feel better buddy!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-25 16:03:16

Thanks 🙂 Pretty sure its just post-conference grossness 😛

Drew Hudson-Viles

2024-03-25 16:05:44

hahah probably!

mboersma

2024-03-25 16:11:20

Feel better Marcus! See you both in a couple weeks.

👍 Drew Hudson-Viles

jsturtevant

2024-03-25 16:42:35

cone there is a PR to remove Virtual Box support. It was used by sig-windows for awhile but is not longer being used or updated. In effort to clean up image-builder we will be removing it unless anyone objects. Will release the PR in 2 weeks. Thanks! cc: @Amim Knabben

👍 Amim Knabben, Marcus Noble, Drew Hudson-Viles, mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:14:49

Those building images should be aware of this and check your recent builds to see if they’re vulnerable.

Version 5.6.0 and 5.6.1 of xz contain a backdoor aimed to bypass SSH authentication: https://lwn.net/Articles/967180/

lwn.net

A backdoor in xz

Andres Freund has posted a detailed investigation into a backdoor that was shipped with versions 5.6.0 and 5.6.1 of the xz compression utility. It appears that the malicious code may be aimed at allowing SSH authentication to be bypassed.

Original URL: https://lwn.net/Articles/967180/

Drew Hudson-Viles

2024-03-29 20:15:28

Thanks for posting this - I was going to get around to it shortly but childcare is a thing right now 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:16:25

Yeah. I haven’t looked into the details too much yet as I’m not at a laptop but I know at least the latest stable Flatcar version is unaffected. Not sure about other distros though.

Drew Hudson-Viles

2024-03-29 20:17:49

My understanding of the read I had was that it's unlikely to affect most distros due to it being in newer/pre-release builds, but that doesn't necessarily mean everyone is fine so they should check.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:19:59

But image builder performs an update of all packages as post of the build process

Drew Hudson-Viles

2024-03-29 20:21:21

It does indeed. I guess if it's in the package-manager - which seems possible, it could be an issue. I'm going to build a ubuntu 22.04 shortly and check it out anyway

👍 Marcus Noble

Abhay Krishna Arunachalam

2024-03-29 20:35:07

I have a cluster running Ubuntu 20.04.6 nodes built with image-builder and here's my output:

/# xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Abhay Krishna Arunachalam

2024-03-29 20:35:44

I think this was built March 7

Drew Hudson-Viles

2024-03-29 20:36:16

22.04.4 LTS

xz --version
xz (XZ Utils) 5.2.5
liblzma 5.2.5

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:36:21

20.04 might not have been updated

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:36:29

Oh nice!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:37:02

So at least for Ubuntu as long as people aren’t manually updating packages they should be safe

Drew Hudson-Viles

2024-03-29 20:37:17

I'm pretty certain in that long thread it did mention it was pre-release builds getting it. I need to digest it properly as it was a skim whilst watching In The Night Garden 😄

Drew Hudson-Viles

2024-03-29 20:39:05

I've just updated the package on my bare metal node at home which also runs 22.04 and it seems it's not getting that compromised package (yet).

apt-cache madison xz-utils
  xz-utils | 5.2.5-2ubuntu1 |  jammy/main amd64 Packages

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:40:41

Ok. So Flatcar and Ubuntu seem safe.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:41:01

I know Archlinux has been vulnerable but that’s not something we need to worry about

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:41:16

Not sure about RHEL or CentOS

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:44:29

From RedHat:

Fedora Linux 40 users may have received version 5.6.0, depending on the timing of system updates. Fedora Rawhide

redhat.com

Urgent security alert for Fedora 41 and Fedora Rawhide users

Red Hat Information Risk and Security and Red Hat Product Security learned that the latest versions of the “xz” tools and libraries contain malicious code that appears to be intended to allow unauthorized access.

Original URL: https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:45:17

From Debian:

Right now no Debian stable versions are known to be affected.
Compromised packages were part of the Debian testing, unstable and
experimental distributions, with versions ranging from 5.5.1alpha-0.1
(uploaded on 2024-02-01), up to and including 5.6.1-1.

Drew Hudson-Viles

2024-03-29 20:45:33

phew

Drew Hudson-Viles

2024-03-29 20:45:46

Pretty clear on the whole then. Maybe a couple edge cases

Marcus Noble (k8s@marcusnoble.co.uk)

2024-03-29 20:47:40

I’m not sure about CentOs, Photon or RockyLinux but I’d guess they’re also ok as I don’t think any of those run with bleeding edge packages.

saifeddine Rajhi

2024-03-31 18:44:55

@saifeddine Rajhi has joined the channel

Staerion

2024-04-01 09:06:44

@Staerion has joined the channel

Karine Santos

2024-04-01 22:45:30

Hello everyone!
I'm looking for the image builder code where these steps for creating vms and resources are defined, and I can't find it. Can someone help me?

Screenshot from 2024-04-01 18-45-03.png

Abhay Krishna Arunachalam

2024-04-01 22:54:55

Image-builder Make targets just call the packer build command with the appropriate configuration files/values.

Packer is in charge of creating the VMs for each virtualization platform (vsphere/openstack/ami/nutanix, etc). through its plugins. In this case, the platform is Openstack so the code for creating VMs and resources will be located at . Plugins directly interact with the virtualization platforms through the available SDKs and APIs (in this case, )

Karine Santos

2024-04-02 00:25:54

tks @Abhay Krishna Arunachalam

Karine Santos

2024-04-02 00:28:39

hello everyone!
Could you help me with a question? I'm trying to upload some k8s images, and I'm encountering the following error. Is there a way to find out why the process of detaching the volume from the instance is not occurring?

Screenshot from 2024-04-01 20-28-27.png

jsturtevant

2024-04-02 16:49:31

which provider is this? It looks to be an issue with the provider code

Karine Santos

2024-04-02 17:27:29

hello @jsturtevant
We use the openstack provider. I would need to increase the server state retry (line 5013 to 5016). For the case below, we understand that we need more time for the desired state. It's possible?

Screenshot from 2024-04-02 13-25-58.png

Karine Santos

2024-04-02 17:29:31

The desired behavior is this (here it tries at least 5x):

Screenshot from 2024-04-02 13-28-29.png

Karine Santos

2024-04-02 17:30:20

My question is if I can somehow change (via ansible, etc.) the retry of the instance state

jsturtevant

2024-04-03 18:35:49

Ansible does have a retry on the tasks: would that help?

Kepler SysAdmin

2024-04-05 12:15:59

oh, we kind of are in the same page
what it is happening in my own opinion
the image builder starts to upload the image, but it jumps to fast for the next step
creating/saving/upload an image could take even 10 mins

Kepler SysAdmin

2024-04-08 13:18:40

@Karine Santos
Did you find a solution for this?

Drew Hudson-Viles

2024-04-08 16:02:02

I don't have a solution for this but I can confirm I use the OpenStack builder on an almost weekly basis without any issues.

It looks like your error is similar to what is being seen here:

and here:

How is OpenStack created in your case - if it's using the methods defined above, you may be hitting this bug.

GitHub

Volume attachment is not deleted when deleting instance · Issue #537 · osism/issues

Reproduce: Create instance with volume Delete instance See that volume is still attached to deleted instance and horizon returns an error Rebuilding seems to be affected as well. Error: Unable to r...

Original URL: https://github.com/osism/issues/issues/537

Launchpad

Bug #2019888 “Detach volume from instance using the Compute API ...” : Bugs : OpenStack Nova Compute Charm

Solutions QA noticed 3 tempest failures with "[Detach volume from instance dd0deef9-d340-40d3-a0c7-fc527bbc0360 using the Compute API (HTTP 409)" in nova compute logs, from <https://solutions.qa.canonical.com/v2/testruns/88316a8c-4c9f-4661-8b67-b2feb448df34> 1) tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_attached_volume Traceback (most recent call last): File "/home/ubuntu/snap/fcbtest/43/.rally/verification/verifier-139f735d-d56e-4087-ab...

Original URL: https://bugs.launchpad.net/charm-nova-compute/+bug/2019888

Karine Santos

2024-04-16 23:25:32

Hello everyone! Thank you for all the contributions!
@Kepler SysAdmin This issue has been fixed in the packer plugin 🙌:
BUGFIX:
RELEASE: V.1.1

If you use packer-plugin-openstack, you will need:
1. Add to the image builder to use a specific version of the packer plugin here:

2. Add variable in dependencies to use this config.pkr.hcl file:

GitHub

fix relying on cinder attachment deletion by felixhuettner · Pull Request #103 · hashicorp/packer-plugin-openstack

Previously the rough flow of building an image was as follows (if using a volume): create a volume create a server do things on the server send a detach call for the volume to cinder upload the vo...

Original URL: https://github.com/hashicorp/packer-plugin-openstack/pull/103

GitHub

Release v1.1.1 · hashicorp/packer-plugin-openstack

What's Changed Exciting New Features 🎉 fix relying on cinder attachment deletion by @felixhuettner in #103 Other Changes .gitignore: ignore crash.log by @lbajolet-hashicorp in #99 cleanup github...

Original URL: https://github.com/hashicorp/packer-plugin-openstack/releases/tag/v1.1.1

GitHub

image-builder/images/capi/packer/openstack/config.pkr.hcl at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/openstack/config.pkr.hcl

GitHub

image-builder/images/capi/Makefile at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/Makefile#L524

👍 Kepler SysAdmin

Karine Santos

2024-04-16 23:27:30

By adjusting this, I was able to upload the images!!!

👍 Drew Hudson-Viles

jhon

2024-04-02 16:33:44

@jhon has joined the channel

Danny Bessems

2024-04-03 06:17:14

Hello, is it possible to easily patch the vsphere ova builders to include an extra 1MB disk when building within the cluster-node-image-builder-amd64 container? So that when deploying with VSphereMachineTemplate I can use spec.template.spec.additionalDisksGiB[] straight away, without first having to edit the template through vCenter...

Danny Bessems

2024-04-03 06:52:49

I know that it is defined within packer-node.json; but I´d rather not override the entire file if possible.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 09:16:58

You should be able to provide additional vars files that overlay onto the existing ones in image-builder.
If you mount a json file with the vars you need into the container you can then specify the PACKERVARFILES environment variable to point to that vars file.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 09:14:23

I've made some progress on the issue we were having with Ubuntu ISO URLs 🎉
It turns out that Ubuntu does have some stable URLs that redirect to old-releases in some situations where needed. I've updated the issue with all the details and I've opened the PR below to update all the ISO URLs we currently reference 🙂 (Except for Ubuntu 23.04 as that currently doesn't have the stable URLs while still in beta 😒)

GitHub

Update Ubuntu 20.04 & 22.04 ISO URLs to use latest stable URLs by AverageMarcus · Pull Request #1438 · kubernetes-sigs/image-builder

Change description This updates all the Ubuntu 20.04 and 22.04 ISO URLs to point to the latest available version using an URL that wont break when a new release is available. Updated 20.04 to late...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1438

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 09:22:19

@Abhay Krishna Arunachalam That update-iso-checksums.sh is fantastic 😄 Made things so easy!

Abhay Krishna Arunachalam

2024-04-05 09:35:04

Glad it's finding some use!

For the Ubuntu 20.04 ones, since we're moving from the legacy server install to the live server install, we need to change the boot command and switch from the debian-installer (d-i) to the subiquity autoinstaller (similar to the 22.04 ones).

Abhay Krishna Arunachalam

2024-04-05 09:40:13

I hadn't included the above boot command and other changes in my original PR that updated the URLs, and it broke our builds when we started consuming IB v0.1.25 in EKS-A, as the VM couldn't find the kickstart file.

So I set out to try and add the necessary changes as a patch on EKS-A, with the intention of contributing the change upstream. I experimented with every combination of boot command sequence possible but couldn't figure it out for the life of me, so ended up reverting the URL changes to Ubuntu 20.04 alone.

GitHub

Update ISO URLs to latest, add script to update ISO checksums by abhay-krishna · Pull Request #1420 · kubernetes-sigs/image-builder

Updating a few ISO URLs pointing to images that got removed from the downloads mirror Bump some Ubuntu ISO URLs to the latest stable point releases Add script to update ISO URL checksums

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1420

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 09:44:19

Ah! I did wonder about those 20.04 ones and if they were different. I thought it might be just different host they were pulled from, didn't realise the actual images were different.
I'll revert those ones and just tackle 22.04 for now.

Abhay Krishna Arunachalam

2024-04-06 01:12:08

Sounds good! Though it'd be nice to eventually align with 22.04 and get rid of the preseed approach entirely as it's been deprecated since 20.04 came out

:plus1: Marcus Noble

Kepler SysAdmin

2024-04-05 09:55:25

@Kepler SysAdmin has joined the channel

Kepler SysAdmin

2024-04-05 10:00:06

Hello there,
I am trying building images, following the instructions on:
but failing on: openstack: Error waiting for image: Resource not found
logs:

==> openstack: Downloading spec file and debug info
    openstack: Downloading Goss specs from, /tmp/goss-spec.yaml and /tmp/debug-goss-spec.yaml to current dir
==> openstack: Stopping server: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
    openstack: Waiting for server to stop: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
==> openstack: Terminating the source server: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
==> openstack: Creating the image: ubuntu-2204
    openstack: Image: c5bc4f59-2440-4094-9720-fa06ae5802b5
==> openstack: Waiting for image ubuntu-2204 (image id: c5bc4f59-2440-4094-9720-fa06ae5802b5) to become ready...
==> openstack: Error waiting for image: Resource not found
==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
==> openstack: Deleted temporary floating IP '71e76a4a-a6e6-4cb2-bc98-9e7c5c6362d3' (178.73.197.24)
==> openstack: Terminating the source server: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
==> openstack: Error terminating server, may still be around: Resource not found
==> openstack: Deleting volume: fdb85e58-1c8a-4d52-bdf2-3fa6720e9b9a ...
==> openstack: Deleting temporary keypair: packer_660f9e6a-fe25-d9cf-6c76-4c2f0149e5d4 ...
Build 'openstack' errored after 10 minutes 8 seconds: Error waiting for image: Resource not found

and the var_file.json I am using:

{
  "source_image": "",
  "networks": "",
  "flavor": "",
  "floating_ip_network": "public",
  "image_name": "ubuntu-2204",
  "image_visibility": "public",
  "image_disk_format": "raw",
  "volume_type": "",
  "ssh_username": "ubuntu",
  "kubernetes_version": "1.28.7"
}

be aware the instance, volume and ssh key gets create it, where it fails is on:

==> openstack: Waiting for image ubuntu-2204 (image id: c5bc4f59-2440-4094-9720-fa06ae5802b5) to become ready...
==> openstack: Error waiting for image: Resource not found

I tried from my laptop, and later from a node that is in the same physical network than the OpenStack Cloud

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 10:20:47

I've not used OpenStack before but are you able to see image c5bc4f59-2440-4094-9720-fa06ae5802b5 in your OpenStack cloud UI at all?

Kepler SysAdmin

2024-04-05 10:24:15

the image isn't there

The time pass so fast, that I am unable to catch it

I am thinking i need to modify some timing between Waiting for image till the next step, because if the image is uploading or creating from a volume, it can last some time around 5-10 mins

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 10:27:37

You could try setting the env var PACKER_LOG=1 to see if you get any more info from the verbose log output.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 10:28:19

Is it possibly a permissions problem? The credentials image-builder running with not having permission to create images? (I'm just guessing here)

Kepler SysAdmin

2024-04-05 10:28:50

the credentials have the right to create images

Kepler SysAdmin

2024-04-05 10:29:01

but I will try: PACKER_LOG=1

🤞 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 10:31:11

Is it possible you're facing this issue:

Any errors in the glance api?

Stack Overflow

cant' make openstack image using packer

I'm trying to build a centos image on openstack with packer. For some reason, the build is terminated in the middle of the script and I can't figure out problems. Additionally, I can't find out any

Original URL: https://stackoverflow.com/questions/51550775/cant-make-openstack-image-using-packer

Kepler SysAdmin

2024-04-05 10:32:35

thx, I need to dig into this, once I found a solution, I will let you know
Thank you again

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 10:32:53

No worries 🙂 Hope you manage to track it down

Kepler SysAdmin

2024-04-05 10:51:51

logs:

# kubectl -n openstack logs pod/glance-api-76759946d4-g9d4n -f | grep 72a7187e-43fa-425b-a52a-a56489e04e6d
Defaulted container "glance-api" out of: glance-api, init (init), glance-perms (init), ceph-keyring-placement (init)
[pid: 14|app: 0|req: 6057/48488] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:31 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 44 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 7|app: 0|req: 6058/48489] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:33 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 44 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 10|app: 0|req: 6067/48490] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:35 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 53 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 8|app: 0|req: 6057/48493] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:40 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 35 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
2024-04-05 09:39:42.286 13 ERROR glance_store._drivers.rbd [None req-65915cac-69f6-45bf-9c50-891063151686 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 72a7187e-43fa-425b-a52a-a56489e04e6d Store Exception unable to receive chunked part: OSError: unable to receive chunked part
[pid: 13|app: 0|req: 6063/48494] 10.0.1.213 () {40 vars in 1101 bytes} [Fri Apr  5 09:39:42 2024] PUT /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d/file => generated 228 bytes in 582 msecs (HTTP/1.1 500) 4 headers in 184 bytes (1 switches on core 0)
[pid: 11|app: 0|req: 6064/48495] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:42 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 45 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 14|app: 0|req: 6058/48496] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:44 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 31 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 10|app: 0|req: 6068/48498] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr  5 09:39:47 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 139 bytes in 16 msecs (HTTP/1.1 404) 4 headers in 164 bytes (1 switches on core 0)

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 10:52:31

Failed to store image 72a7187e-43fa-425b-a52a-a56489e04e6d Store Exception unable to receive chunked part: OSError: unable to receive chunked part

What an incredibly unhelpful error 😞

Kepler SysAdmin

2024-04-05 10:53:04

just changing the logs from info to debug, and run the image-build again

Kepler SysAdmin

2024-04-05 10:53:14

that will take a couple of mins

Kepler SysAdmin

2024-04-05 11:39:16

2024-04-05 10:33:26.737 9 ERROR glance.common.wsgi OSError: unable to receive chunked part

Kepler SysAdmin

2024-04-05 11:42:48

if you wonder, this is after:

[pid: 9|app: 0|req: 24/165] 10.0.1.213 () {40 vars in 1100 bytes} [Fri Apr  5 10:33:26 2024] PUT /v2/images/796d1a83-a325-43af-9c95-f1e098f1467b/file => generated 228 bytes in 636 msecs (HTTP/1.1 500) 4 headers in 184 bytes (1 switches on core 0)
/var/lib/openstack/lib/python3.10/site-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: unknown. To ensure interoperability, identifiers should be a valid uuid.
  warnings.warn(('Invalid uuid: %s. To ensure interoperability, '
[pid: 14|app: 0|req: 21/166] 188.78.244.135 () {44 vars in 909 bytes} [Fri Apr  5 10:33:27 2024] GET /v2/images/796d1a83-a325-43af-9c95-f1e098f1467b => generated 957 bytes in 28 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
2024-04-05 10:33:27.792 11 WARNING oslo_policy.policy [None req-20b22860-76c5-4a3a-b592-8ca10b062eed 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use oslopolicy-convert-json-to-yaml tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way:

could be an issue with the current version of OpenStack (. .. maybe ) many deprecated operations from several versions still work on new versions

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 11:43:28

Yeah I think this is an issue on the OpenStack side and not image-builder. 😞

Kepler SysAdmin

2024-04-05 11:44:22

Do you know what current k8s versions been supported in the image-builder?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 11:45:49

You should be able to override the version in the vars you provide. It should be compatible up to v1.29

👍 Kepler SysAdmin

Kepler SysAdmin

2024-04-05 12:00:33

@Marcus Noble
by the way,
the image start to be upload it:

==> openstack: Waiting for image ubuntu-2204 (image id: 1d34cf8f-ae60-4181-85e0-f0d549d5142c) to become ready...
2024/04/05 12:58:09 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:09 Waiting for image creation status: queued
2024/04/05 12:58:11 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:11 Waiting for image creation status: queued
2024/04/05 12:58:14 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:14 Waiting for image creation status: queued
2024/04/05 12:58:16 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:16 Waiting for image creation status: queued
2024/04/05 12:58:18 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:18 Waiting for image creation status: saving
2024/04/05 12:58:21 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:21 Waiting for image creation status: queued
2024/04/05 12:58:23 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:23 Waiting for image creation status: queued
==> openstack: Error waiting for image: Resource not found

but it does not last, probably the builder goes to the next step too fast

Kepler SysAdmin

2024-04-05 12:16:53

I think Karine is having the same issue:
https://kubernetes.slack.com/archives/C01E0Q35A8J/p1712014119475089

Drew Hudson-Viles

2024-04-08 16:12:22

Just to follow up here as well (been out of the loop for a few days unfortunately) I can confirm I use OpenStack remote on an almost weekly basis and I don't have any issues with it functioning in terms of creating an image at the end of a build. As a result I'd suspect this is an OpenStack config issue rather than an image builder one.

Are you able to use the same OS credentials to manually upload an image using the OpenStack cli?

Kepler SysAdmin

2024-04-08 16:12:52

"Are you able to use the same OS credentials to manually upload an image using the OpenStack cli?"
Yes

Drew Hudson-Viles

2024-04-08 16:13:12

Which version of OpenStack are you running out of interest?

Drew Hudson-Viles

2024-04-08 16:13:27

and how is it deployed - IE Kolla, manually using services etc?

Kepler SysAdmin

2024-04-08 16:14:11

some fresh logs:

==> openstack: Terminating the source server: e68cfe5f-390c-4c5f-b895-861d3f0d47bd ...
2024/04/08 15:12:16 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:16 Waiting for state to become: [DELETED]
2024/04/08 15:12:16 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:16 Waiting for state to become: [DELETED] currently SHUTOFF (0%)
2024/04/08 15:12:19 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:19 [INFO] 404 on ServerStateRefresh, returning DELETED
==> openstack: Creating the image: ubuntu-2204
    openstack: Image: 89fdd79b-bf9b-42f8-9b69-c245ce53f945
==> openstack: Waiting for image ubuntu-2204 (image id: 89fdd79b-bf9b-42f8-9b69-c245ce53f945) to become ready...
2024/04/08 15:12:20 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:20 Waiting for image creation status: queued
2024/04/08 15:12:22 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:22 Waiting for image creation status: queued
2024/04/08 15:12:24 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:24 Waiting for image creation status: queued
2024/04/08 15:12:26 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:26 Waiting for image creation status: queued
2024/04/08 15:12:28 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:28 Waiting for image creation status: queued
2024/04/08 15:12:30 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:30 Waiting for image creation status: saving
2024/04/08 15:12:32 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:32 Waiting for image creation status: queued
2024/04/08 15:12:34 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:34 Waiting for image creation status: queued
2024/04/08 15:12:36 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:36 Waiting for image creation status: queued
==> openstack: Error waiting for image: Resource not found
==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
==> openstack: Deleted temporary floating IP '7c1a29c8-1767-4610-8a17-eba3e7cf7c14' (178.73.197.119)
==> openstack: Terminating the source server: e68cfe5f-390c-4c5f-b895-861d3f0d47bd ...
==> openstack: Error terminating server, may still be around: Resource not found
==> openstack: Deleting volume: c5805931-8a3b-42a7-851f-d444d9cd5bef ...
==> openstack: Deleting temporary keypair: packer_66140781-ae67-d1d3-0d7d-9bb69ca0a3c3 ...
2024/04/08 15:12:39 [INFO] (telemetry) ending openstack
==> Wait completed after 8 minutes 6 seconds
2024/04/08 15:12:39 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/04/08 15:12:39 machine readable: openstack,error []string{"Error waiting for image: Resource not found"}
==> Builds finished but no artifacts were created.
Build 'openstack' errored after 8 minutes 6 seconds: Error waiting for image: Resource not found

Which version of OpenStack are you running out of interest?
Let me find out

Kepler SysAdmin

2024-04-08 16:14:44

openstack helm charts ( atmosphere from Vexxhost )

Kepler SysAdmin

2024-04-08 16:18:50

"Which version of OpenStack are you running out of interest?"
Bobcat
27.0.1 for glance

Kepler SysAdmin

2024-04-08 16:25:00

if you wonder about glance logs:

# kubectl -n openstack logs pod/glance-api-bd89db7b4-5vjvv | grep 89fdd79b-bf9b-42f8-9b69-c245ce53f945 | grep -i error
Defaulted container "glance-api" out of: glance-api, init (init), glance-perms (init), ceph-keyring-placement (init)
2024-04-08 15:12:30.390 14 ERROR glance_store._drivers.rbd [None req-f045c8b7-f9ea-4ccb-9fea-3baed1a680dc 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 89fdd79b-bf9b-42f8-9b69-c245ce53f945 Store Exception unable to receive chunked part: OSError: unable to receive chunked part
2024-04-08 15:12:33.695 12 ERROR glance_store._drivers.rbd [None req-f045c8b7-f9ea-4ccb-9fea-3baed1a680dc 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 89fdd79b-bf9b-42f8-9b69-c245ce53f945 Store Exception unable to receive chunked part: OSError: unable to receive chunked part
2024-04-08 15:12:35.242 9 ERROR glance_store._drivers.rbd [None req-f045c8b7-f9ea-4ccb-9fea-3baed1a680dc 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 89fdd79b-bf9b-42f8-9b69-c245ce53f945 Store Exception unable to receive chunked part: OSError: unable to receive chunked part

Drew Hudson-Viles

2024-04-08 16:30:21

ok that's a new approach to me so I'm not sure how it's configured out the gate however I know they contributed to my PR which introduced the OpenStack remote approach - I've only got experience with Kolla (and I have infra people who actually spin it up, I just interact with it).

OK, so it works via the CLI - same credential and endpoints etc and it's just the packer approach that's failing - as it has been mentioned this isn't actually image builder itself because by this point it's the OpenStack packer plugin doing the work.

It may be worth reaching out to them in the issues of their helm repo as it could be a simple configuration change that may solve this.

Kepler SysAdmin

2024-04-08 16:31:53

"It may be worth reaching out to them in the issues of their helm repo as it could be a simple configuration change that may solve this"
Who Hashicorp or Vexxhost?

Drew Hudson-Viles

2024-04-08 16:35:20

Vexxhost as it's their chart

Kepler SysAdmin

2024-04-08 16:38:08

Theese are the configs:

  conf:
    glance:
      DEFAULT:
        log_config_append: null
        show_image_direct_url: true
        show_multiple_locations: true
        enable_import_methods: "[]"
        workers: 8
      cors:
        allowed_origins: "**"
      image_format:
        disk_formats: "qcow2,raw"
      oslo_messaging_notifications:
        driver: noop

I can change those, let me know

Kepler SysAdmin

2024-04-08 16:40:11

{
  "source_image": "",
  "networks": "",
  "flavor": "",
  "floating_ip_network": "public",
  "image_name": "ubuntu-2204",
  "image_visibility": "public",
  "image_disk_format": "raw",
  "volume_type": "",
  "ssh_username": "ubuntu",
  "kubernetes_version": "1.28.7"
}

Is it something to do with the image disk format?

Drew Hudson-Viles

2024-04-08 16:47:46

The image disk format should be fine - this is what I use in my build.

{
  "kubernetes_cni_semver": "v1.3.0",
  "kubernetes_cni_deb_version": "1.3.0-1.1",
  "crictl_version": "1.29.0",
  "kubernetes_semver": "v1.29.2",
  "kubernetes_series": "v1.29",
  "kubernetes_deb_version": "1.29.2-1.1",
  "extra_debs": "nfs-common",
  "image_name": "",
  "source_image": "",
  "networks": "",
  "flavor": "",
  "attach_config_drive": "true",
  "use_floating_ip": "true",
  "floating_ip_network": "",
  "security_groups": "",
  "image_visibility": "public",
  "image_disk_format": "raw",
  "use_blockstorage_volume": "true",
  "volume_type": "",
  "volume_size": "12",
  "qemu_binary": "",
  "disk_size": "",
  "output_directory": ""
}

As for the configuration I can't advise on that from my side as I've never used Vexxhost's helm charts - I'd recommend having a look at any ingress logs (presuming it deploys them) too as they may reveal some information as to why chunks are failing. The bit that's making me think it's either an OpenStack or at the highest level, a helm chart issue is the OSError: unable to receive chunked part error you're seeing.

👍 Kepler SysAdmin

Kepler SysAdmin

2024-04-08 16:50:40

Thanks

Kepler SysAdmin

2024-04-08 17:09:49

it looks like there is fix by them

trying the above version

GitHub

fix(uwsgi): tune settings by mnaser · Pull Request #1047 · vexxhost/atmosphere

The current uWSGI settings that we ship are missing a few important ones, specifically http-raw-body which is used by Glance in order to enable chunked uploads. This is the reason why a direct uplo...

Original URL: https://github.com/vexxhost/atmosphere/pull/1047

👍 Drew Hudson-Viles

Kepler SysAdmin

2024-04-08 17:17:53

It looks like is working
2024/04/08 16:17:19 packer-plugin-openstackv1.1.2x5.0linuxamd64 plugin: 2024/04/08 16:17:19 Waiting for image creation status: saving
2024/04/08 16:17:21 packer-plugin-openstackv1.1.2x5.0linuxamd64 plugin: 2024/04/08 16:17:21 Waiting for image creation status: saving
2024/04/08 16:17:23 packer-plugin-openstackv1.1.2x5.0linuxamd64 plugin: 2024/04/08 16:17:23 Waiting for image creation status: saving

Kepler SysAdmin

2024-04-08 17:18:02

It looks like is working:

2024/04/08 16:17:19 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 16:17:19 Waiting for image creation status: saving
2024/04/08 16:17:21 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 16:17:21 Waiting for image creation status: saving
2024/04/08 16:17:23 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 16:17:23 Waiting for image creation status: saving

Drew Hudson-Viles

2024-04-08 17:18:20

Nice. Fingers crossed! 🤞🤞

Kepler SysAdmin

2024-04-08 17:23:04

It worked!

Drew Hudson-Viles

2024-04-08 17:33:22

Awesome!! Glad it has

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 15:20:28

@mboersma Does CAPZ currently use image-builder to build Flatcar based images and make them available for others to use?

mboersma

2024-04-05 15:56:01

You can build Flatcar images for Azure with image-builder (as you know), but the CAPZ team only publishes reference images for Ubuntu and Windows to the Azure Marketplace.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 15:56:20

Right, nothing I can copy then 😆

I find the whole image gallery / image templates / marketplace images thing in Azure really confusing 😅

👍 mboersma

mboersma

2024-04-05 15:57:47

We want to switch to publishing just to shared image galleries, which would be somewhat simpler (and is also our recommended path for users in our docs). But not there quite yet.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 15:59:21

Yeah, that's what we're doing. 🙂
I'm in the process of switching our source image from being one from another shared gallery to a marketplace image but I was having trouble with it being blocked because billing reasons.
I've finally found the list of approved images which I think solves my problem. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 16:00:09

I'm just now confused as to why Flatcar has 4 different offers on the marketplace and what the difference is between them (see )

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)

Hey y'all 👋 I was just wondering, is there any documentation anywhere that explains the 4 different Flatcar offers available on the Azure Marketplace? <code>az vm image list --all -p kinvolk -f flatcar | jq '[.[].offer] | unique' [ "flatcar-container-linux", "flatcar-container-linux-corevm", "flatcar-container-linux-corevm-amd64", "flatcar-container-linux-free" ]</code> The <a href="https://www.flatcar.org/docs/latest/installing/cloud/azure/">Running on Microsoft Azure</a> docs mention <code>flatcar-container-linux</code> and <code>flatcar-container-linux-free</code> but not the <code>corevm</code> ones. If I understand correctly from the <a href="https://github.com/Azure/azure-compute-gallery/blob/main/approved-community-gallery-images.csv">approved community gallery images list</a> the <code>corevm</code> ones are the only ones allowed to be used to then build images into community galleries. (I did come across <a href="https://github.com/flatcar/Flatcar/issues/872">this issue</a> that offered a bit of insight but not completely) Is there any difference between the images built in these different offers?

Original URL: https://kubernetes.slack.com/archives/C03GQ8B5XNJ/p1712328741483559

mboersma

2024-04-05 16:07:37

I'm really not sure--the Flatcar team does their own publishing (although not specifically for Cluster API).

I can dig around and try to find out. Also maybe @Mateusz Gozdek (invidian) or @Jeremi Piotrowski knows?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 16:09:01

There's no rush. I've just successfully built and published based on the corevm offer. I just need to test it out now to make sure it actually works 😆
It would be nice to know if there are differences though.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 16:20:04

What's the difference between an Image and a VM image definition?
Image-builder produces both but I'm not sure what is used for what 😕

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 16:50:44

Ah this explains it:
Image is the source that is used to create the version in the image definition 🙂

learn.microsoft.com

Create an image definition and image version - Azure Virtual Machines

Learn how to create an image in an Azure Compute Gallery.

Original URL: https://learn.microsoft.com/en-us/azure/virtual-machines/image-version?tabs=portal%2Ccli2

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 16:53:55

I'd like to check some of my understanding with Azure...

With image-builder we create a "managed image" from the VM in our subscription . When the destination is a community gallery this "managed image" is used to create the "version" within the Image definition of our gallery. Is that correct?

If so, do we still need the "managed image" after we've published the version to the gallery? If its safe to delete, is there a way to do this with image-builder?

mboersma

2024-04-05 17:17:23

I think your understanding is correct. As far as I can tell (mostly by having read the docs, not actually experimented), you can create image versions and then delete the managed image since the version is used to provision.

I found this post too that suggests you can do this but it might prevent expanding replication to other regions? Or maybe not. .

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 17:19:03

That's fine, we handle the replication as part of the image-builder run. As long as we don't want extra regions after it's built we're ok. (and we could always just re-build if that was the case)

Now, any idea if image-builder / Packer can automatically clean this up?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 17:20:29

I did come across this issue which seems to suggest that it is meant to be removing the managed image and was broken at one point but then fixed. But I dont see this happening in my environment 😞

GitHub

Packer not cleaning up. Azure Image Gallery · Issue #9615 · hashicorp/packer

Overview of the Issue Packer leaves behind an interim image (managed_image_name) and sometimes VM resources. Seems to be related to using a pre-existing Azure resource group instead of the auto-gen...

Original URL: https://github.com/hashicorp/packer/issues/9615

mboersma

2024-04-05 17:21:09

I'm looking at the docs for the ARM builder and don't see anything that looks relevant to cleaning the managed image.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 17:21:33

Yeah, same 😞

mboersma

2024-04-05 17:22:02

Also in image-builder we're still using the 1.x version of the Azure packer builder plugin, not 2.0 yet.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 17:22:39

Oh. I wonder if that could be part of it 🤔

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 17:22:57

This pull request first appeared in v1.7.1

mboersma

2024-04-05 17:23:02

Because we still need support for VHD publishing with SAS URLs which I think 2.x removes.

mboersma

2024-04-05 17:26:18

The plugin is up to 2.0.5 now:
Upgrading requires CAPZ to move to community gallery publishing IMHO, which is something I'm working on but not there yet.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-05 17:27:32

Ok. It's not a huge problem. I was just trying to figure out what was going on in our account. I'm not too bothered with the Images being left behind. Maybe when we finally upgrade they'll magically start being cleaned up and I don't need to think about it 😄

mboersma

2024-04-05 17:27:59

One can hope!

Slackbot

2024-04-08 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-08 17:05:19

⚠️ Currently all PRs are failing due to an issue with photon-4 packages not matching known keys. This is causing the pull-ova-all tests to fail for everyone.

If anyone happens to know what might have happened to cause this and how to fix it that's would be a huge help. If we don't track down what the problem is in the next couple days we'll remove the photon-4 OS from the test to unblock PRs while we figure out the problem. (Note, this doesn't seem to effect photon 3 or 5)

Issue:

GitHub

Photon 4 causing `pull-ova-all` to fail on all PRs · Issue #1441 · kubernetes-sigs/image-builder

Environment N/A - This is happening on the pull-ova-all Prow job on all PRs. Example PRs: #1438 #1439 What steps did you take and what happened? Run /test pull-ova-all on a PR and the following err...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1441

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-11 13:17:51

I'd like to better understand how Packer make use of SSH keys when building image with image-builder and I have a couple questions I'm not clear on:

What is this SSH key? It has the associated user as capi but the name property in the users list is capv 😕 If I'm following correctly this is used in image-post-create-config.sh which is then used by the OVA provider which makes the capv make sense. Not sure if those files in cloudinit are used by any other providers though. Can we remove this ssh key?

Is there any reason for us not to add sshclearauthorizedkeys (docs) to image-builder so that the ~/.ssh/authorizedkeys file is emptied before we publish the images?

Paramita

2024-04-15 13:06:34

@Paramita has joined the channel

Anubhav Gain

2024-04-16 04:44:31

@Anubhav Gain has joined the channel

Abhay Krishna Arunachalam

2024-04-17 08:17:44

Hi image-builder maintainers! I wanted to know if there's a tentative date for the next release of image-builder v0.1.26.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-17 08:25:33

I want to get https://github.com/kubernetes-sigs/image-builder/pull/1438 merged in first now that it’s unblocked by the failing test then I think we should be good to go.

GitHub

Update Ubuntu 22.04 ISO URLs to use latest stable URLs by AverageMarcus · Pull Request #1438 · kubernetes-sigs/image-builder

Change description This updates all the Ubuntu 22.04 ISO URLs to point to the latest available version using an URL that wont break when a new release is available. Updated 22.04 to latest: 22.04....

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1438

Abhay Krishna Arunachalam

2024-04-17 08:29:16

Nice, thanks! yay-fox

xinity

2024-04-19 11:59:31

@xinity has joined the channel

xinity

2024-04-19 12:03:27

hello there, back into capi world i'm trying to build flatcar qemu image from the repository but failing miserably 😞 the latest log i'm having is:

2024/04/19 10:52:11 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:11 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 [INFO] Attempting SSH connection to 127.0.0.1:2686...
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 [DEBUG] reconnecting to TCP connection for SSH
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 [DEBUG] handshaking with SSH
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 Keyboard interactive challenge:
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 -- User:
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 -- Instructions:
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 -- Question 1: Password:
2024/04/19 10:52:20 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:20 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey keyboard-interactive], no supported methods remain

tested with plain ssh connection and same behavior 😞

anyone may have a clue to fix this ?

Jeremi Piotrowski

2024-04-19 13:59:29

what is your host ssh version?

xinity

2024-04-19 14:05:19

8.7p1-34.el9.x86_64

xinity

2024-04-19 14:05:33

i'm trying to build on rockylinux9

Jeremi Piotrowski

2024-04-19 14:10:00

thats a very old ssh version and you may have protocol issues because of that

Jeremi Piotrowski

2024-04-19 14:10:05

check this commit: 74f31d3b4b5b7fb12fd3340f6fb35d9cfdc26d67

Jeremi Piotrowski

2024-04-19 14:10:25

or i suggest running the build from a docker container with a more recent ssh client

xinity

2024-04-19 14:15:40

how would you build from a docker container ? that's an interesting point of view

Jeremi Piotrowski

2024-04-19 14:16:37

there is a Dockerfile in images/capi

Jeremi Piotrowski

2024-04-19 14:16:50

you can use that as a build environment

Jeremi Piotrowski

2024-04-19 14:17:12

if you need qemu then run privileged or pass through /dev/kvm

❤️ xinity

xinity

2024-04-19 14:25:15

will try with the docker build mechanism

xinity

2024-04-19 14:25:17

will keep you posted

xinity

2024-04-19 20:53:44

tried finally with the docker image, same issue @Jeremi Piotrowski 😢

xinity

2024-04-20 12:22:44

i've opened a bug report :

GitHub

flatcar qemu build broken: ssh connection failed · Issue #1445 · kubernetes-sigs/image-builder

Environment Make target: make build-qemu-flatcar Run using container image? (Y/N): Y (also tried with pure cli on the host) Environment vars: None used Vars file: None used What steps did you take ...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1445

:thank_you_icon: Marcus Noble

Mike Tritabaugh

2024-04-19 21:39:31

@Mike Tritabaugh has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-04-22 11:52:51

There's currently nothing on the agenda for todays image-builder office hours so unless someone has anything they'd like to bring up I suggest we skip for this week.

I do think it's time for us to do a new release though now that we're unblocked by the failing tests so if anyone has any PRs they'd like to get merged in before we do the next release please speak up now 🙂

mboersma

2024-04-22 15:24:59

+1 from me on both points.

👍 Marcus Noble

Slackbot

2024-04-22 15:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2024-04-22 16:32:44

No topics on the list, so we skipped today. See you on May 6th!

Kishore

2024-04-30 07:25:16

@Kishore has joined the channel

Kishore

2024-04-30 07:40:00

Hi,

I am creating windows image for OCI. Below is the PACKERVARFILES specification. But not able to build Windows image. Is there anything missed in this configuration.

{
"buildname": "windows",
"baseimageocid": "ocid1.image.oc1.eu-frankfurt-1.aaaaaaaaiwwre36icxfiivmgqlfdbrjm67igscbikjq4k2luhbjgcwyxiywa",
"ocpus": "128",
"shape": "BM.Standard.E4.128",
"region": "eu-frankfurt-1",
"compartmentocid": "ocid1.compartment...",
"subnetocid": "ocid1.subnet.oc1..",
"availabilitydomain": "DDJb:EU-FRANKFURT-1-AD-1",
"userocid": "ocid1.user..",
"fingerprint": "af:66:ce:6b:63:d1:ef:99:97:43:50:36:35:f2:71:f9",
"tenancyocid": "ocid1.tenancy....",
"keyfile": "~/.oci/ociapikey.pem"
}
/usr/bin/packer build -var-file="/home/ubuntu/image-builder/images/capi/packer/config/kubernetes.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/kubernetes.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/containerd.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/containerd.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/ansible-args-windows.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/common.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/common.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/cloudbase-init.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/goss-args.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/additionalcomponents.json" -color=true -var-file="/home/ubuntu/image-builder/images/capi/packer/oci/windows-2022.json" -var-file="/home/ubuntu/image-builder/images/capi/oci.json" packer/oci/packer-windows.json
Error: Failed to prepare build: "oracle-oci"

3 error(s) occurred:

* 'fingerprint' must be specified
* 'securitytokenfile' must be correctly specified. did not find a proper
configuration for key id
** 'key_file' must be correctly specified. did not find a proper configuration
for private key

Kishore

2024-04-30 09:04:06

Got resolved. Added missing items in packer-windows.json

🎉 Marcus Noble, mboersma, Drew Hudson-Viles

georgeb

2024-04-30 12:41:42

@georgeb has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-05 19:28:26

It’s a public holiday in the UK tomorrow so I’ll be skipping the office hours. There’s currently nothing on the agenda so feel free to skip if nothing comes up. 🙂

☝️ Drew Hudson-Viles, mboersma

Slackbot

2024-05-06 15:30:15

Reminder: Image-Builder office hours start in 1 hour. Agenda:

xinity

2024-05-10 09:03:29

any idea why when i built a flatcar image , the ssh pub key isn't populated when used with CAPI ?

Kepler SysAdmin

2024-05-14 13:33:20

Hello All,
the image-builder works wonderfully,
Now, I know this is just for build images with pre-built k8s.
Is there something similar but with pre-built wordpress?

voor

2024-05-14 13:35:05

You would deploy a kubernetes cluster and then run WordPress. https://bitnami.com/stack/wordpress

bitnami.com

WordPress

WordPress is one of the world’s most popular web publishing platforms for building blogs and websites. It can be customized via a wide selection of themes, extensions and plug-ins.

Original URL: https://bitnami.com/stack/wordpress

Kepler SysAdmin

2024-05-14 13:38:45

Hi....
yes, but i need this as a openstack image
already have it as on a k8s cluster

voor

2024-05-14 16:02:43

If you need an openstack image you could look at

Or you could use packer directly for openstack

Medium

Using Bitnami Images with Openstack

In an OpenStack cloud, deploying Bitnami configurations can be even more quickly achieved. We are running OpenStack Icehouse and KVM or…

Reading time

3 min read

Original URL: https://medium.com/@soumok.dutta.cyberliterate/using-bitnami-images-with-openstack-22d56014cca1

developer.hashicorp.com

OpenStack Builder | Integrations | Packer | HashiCorp Developer

Explore Packer product documentation, tutorials, and examples.

Original URL: https://developer.hashicorp.com/packer/integrations/hashicorp/openstack/latest/components/builder/openstack

syt

2024-05-17 02:49:09

@syt has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 09:06:03

I was planning to cut a new release today as it's been a while since the last and we've had quite a few PRs merged since then. Any PRs that people would like to try and get in before I do?

Drew Hudson-Viles

2024-05-20 09:12:24

I'm good from my side, thanks!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 13:19:43

ok, I'm going to start a new release now.

👍 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 13:37:12

GitHub

Image promotion for scl-image-builder v0.1.26 by AverageMarcus · Pull Request #6831 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.26 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers "@AverageMarcus @jsturtevant @kkes...

Original URL: https://github.com/kubernetes/k8s.io/pull/6831

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 13:50:59

Docs PR:

GitHub

Updated docs for v0.1.26 by AverageMarcus · Pull Request #1471 · kubernetes-sigs/image-builder

Change description Related issues Fixes # Additional context

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1471

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 13:55:46

@Drew Hudson-Viles

tide Pending — Not mergeable. Needs approved, lgtm labels.

Drew Hudson-Viles

2024-05-20 13:56:19

Sorry, did a silly and forgot the comment when approving 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 13:56:33

It's ok. I find it weird that lgtm is still needed with approve 😆

☝️ Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 14:00:20

laugh-cry

tide Pending — Not mergeable. Needs approved label.

Drew Hudson-Viles

2024-05-20 14:01:04

Damn you automation! 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 14:01:18

shake_fist

Drew Hudson-Viles

2024-05-20 14:01:27

So even though I approved, I needed to do /approve... nice, nice. 😄

Drew Hudson-Viles

2024-05-20 14:01:46

All done buddy

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 14:02:04

yeah, tide doesn't use the GitHub PR status. It keeps its own state based on comments / labels

Drew Hudson-Viles

2024-05-20 14:02:13

That's fair 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 14:02:21

It... frustrating sometimes 😆

😆 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 13:52:04

Image-builder v0.1.26 is now available:
Thanks to all contributors! 🎉

GitHub

Release v0.1.26 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.26">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.26</a> What's Changed docs: Updated docs for 0.1.25 release by @dre...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.26

🎉 Drew Hudson-Viles, Abhay Krishna Arunachalam, jsturtevant, Mitchel Haring

Abhay Krishna Arunachalam

2024-05-21 05:36:21

Nice!

Slackbot

2024-05-20 15:30:26

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 15:48:49

@mboersma do you still want to talk about Ubuntu 24.04?

mboersma

2024-05-20 15:57:14

Sure, we can move that to today, but it's a short topic.

👍 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-20 16:35:07

@Drew Hudson-Viles are you joining? You have an item on the agenda

Drew Hudson-Viles

2024-05-20 16:35:25

I am, I'll be 2 minutes, just on a call.

Sriraman Srinivasan

2024-05-21 04:23:43

Do we have a recording of this?

Sriraman Srinivasan

2024-05-21 04:28:03

Sorry found the link in the doc

👍 mboersma

Sriraman Srinivasan

2024-05-21 04:25:49

Just wondering about the impact of this in IB

GitHub

stop shelling out to crictl from kubeadm · Issue #3064 · kubernetes/kubeadm

instead of shelling out to crictl for CRI actions kubeadm can use the new <https://git.k8s.io/cri-client> library. actions 1.31 switch kubeadm to use the cri-client and only show a warning if crictl ...

Original URL: https://github.com/kubernetes/kubeadm/issues/3064

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 15:29:13

@mboersma I think this change has broken Flatcar on AWS 😞

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 15:29:28

    amazon-ebs.flatcar-stable: TASK [providers : Install AWS CLI v2] ************************************************************************************
    amazon-ebs.flatcar-stable: fatal: [default]: FAILED! => {"changed": true, "cmd": ["/tmp/aws/install", "-i", "/usr/local/aws-cli", "-b", "/usr/local/sbin"], "delta": "0:00:01.552103", "end": "2024-05-23 14:19:56.504588", "msg": "non-zero return code", "rc": 1, "start": "2024-05-23 14:19:54.952485", "stderr": "mkdir: cannot create directory '/usr/local/aws-cli': Read-only file system", "stderr_lines": ["mkdir: cannot create directory '/usr/local/aws-cli': Read-only file system"], "stdout": "", "stdout_lines": []}
    amazon-ebs.flatcar-stable:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 15:29:49

What was the reason for removing these aws cli changes specifically?

mboersma

2024-05-23 15:34:44

Oops! It was to support Ubuntu 24.04 on AWS, I unified the couple of paths that used to be there to install the AWS CLI (and added gpg checksum validation). I didn't realize that would pull in Flatcar actually.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 15:36:09

I'm going to try and get a PR up.

mboersma

2024-05-23 15:37:27

Thanks Marcus! LMK if you need help or want me to fix my own damage, I'm just waking up but I'd be glad to look into it.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 15:37:50

I think I have it already. Haven't tested it yet but I'll get the PR up for you to take a look at

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 15:39:36

GitHub

Fix aws cli installation on Flatcar by AverageMarcus · Pull Request #1472 · kubernetes-sigs/image-builder

Change description Fixes installation of aws-cli on Flatcar. Related issues No issue but currently building Flatcar for CAPA results in the following error: amazon_ebs.flatcar_stable: TASK [provi...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1472

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-23 16:04:25

Confirmed working for Flatcar

👍 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 07:09:18

I'm going to get a new release out with this fix today

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 07:51:46

GitHub

Image promotion for scl-image-builder v0.1.27 by AverageMarcus · Pull Request #6842 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.27 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers "@AverageMarcus @jsturtevant @kkes...

Original URL: https://github.com/kubernetes/k8s.io/pull/6842

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 08:14:13

And then once done I have this for the docs:

GitHub

Update docs for v0.1.27 by AverageMarcus · Pull Request #1473 · kubernetes-sigs/image-builder

Updates the docs for the new release v0.1.27 Also fixed a bug in the release-update-docs script that incorrectly got the date of the previous tag. /hold for new release published

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1473

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 09:15:50

@Drew Hudson-Viles as you're about would you mind looking at the above? 🙏 🥺

Drew Hudson-Viles

2024-05-24 09:16:16

Sure can, hold please!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 09:16:28

My hero! hero

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 09:19:34

(and the one above it 😉 )

Drew Hudson-Viles

2024-05-24 09:19:42

yeah I'm on it now 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 09:19:49

💙

Drew Hudson-Viles

2024-05-24 09:20:19

all done 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 09:21:01

Thank you kindly sir 🙂

:gr_captain_salute: Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-24 09:31:01

Image-builder v0.1.27 is now available:

This is a small release with just a couple bug fixes in it. 🙂

Thanks to all contributors!

GitHub

Release v0.1.27 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.27">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.27</a> What's Changed Updated docs for v0.1.26 by @AverageMarcus in...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.27

:emoji_k8s_loft: Drew Hudson-Viles, jsturtevant, Abhay Krishna Arunachalam, Mitchel Haring, mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-27 18:32:34

Image-builder v0.1.28 is now available:
Thanks to all contributors!
(Trying to do more frequent, smaller releases. Lets see how this goes 😄)

GitHub

Release v0.1.28 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.28">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.28</a> What's Changed Update docs for v0.1.27 by @AverageMarcus in ...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.28

🎉 Drew Hudson-Viles, Mitchel Haring

Jacob Weinstock

2024-05-28 01:16:01

@Jacob Weinstock has joined the channel

Jonathan Rosser

2024-05-28 15:11:03

@Jonathan Rosser has joined the channel

Sriraman Srinivasan

2024-05-29 07:46:02

With PR for license exception on packer getting rejected, whats the plan going forward?

GitHub

[License Exception Request] Packer for `image-builder` · Issue #625 · cncf/foundation

Original URL: https://github.com/cncf/foundation/issues/625#issuecomment-2127628513

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-29 07:47:47

It's unclear. Please see the discussion in our tracking issue:

For right now nothing changes as we're still pinned to the pre-BUSL version but that can't continue forever.

GitHub

Investigate implications of new Packer licensing · Issue #1246 · kubernetes-sigs/image-builder

HashiCorp recently changed the license on some of their open source projects to BSL 1.1. This may or may not present legal issues to the use of Packer in the image-builder project. Let's communicat...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1246

Marcus Noble (k8s@marcusnoble.co.uk)

2024-05-29 07:48:54

I have also added it to the agenda for next weeks office hours though I'm not sure there will be any solution come out of that. It'll likely be more to make the current state of things clear.

Jonathan Rosser

2024-05-29 14:40:46

i think there is another file to clean up on ubuntu images etc/cloud/cloud.cfg.d/90-installer-network.cfg - see

GitHub

subiquity.network: cloud-init networking when netplan root-readonly · canonical/subiquity@2af5829

When cloudinit.features.NETPLAN_CONFIG_ROOT_READ_ONLY is True, cloud-init will write /etc/netplan/50-cloud-init.yaml as read-only root. This added security allows for subiquity to use cloud-init&#...

Original URL: https://github.com/canonical/subiquity/commit/2af582984c47ebe0a8d3bbc2733e0767d04cda0f

Jonathan Rosser

2024-05-29 14:57:23

i found this by trying to use the 22.04 daily build iso instead of the 22.04.04 release ISO, and you get revision 5495 vs 5741 of the subiquity snap in those different ISOs

Kumar

2024-05-29 22:47:21

@Kumar has joined the channel

Christopher

2024-06-01 09:01:02

@Christopher has joined the channel

Slackbot

2024-06-03 15:30:13

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-06-04 10:10:10

Just catching up on the office hours... some notes:

I spoke to the Flatcar people at KubeCon who are trying to get sysext support added to Ubuntu and other OSs too

I'm happy with Windows being a different approach to others, that's an acceptable split to me

At Giant Swarm we make heavy use of the image-builder container image so we'd need to switch to building our own image (we actually already do with the upstream image-builder one as the base)

If sysext is the way forward for Linux OSs then I'm not sure there's a need for image-builder going forward (which is fine if that is the direction things are going)

I agree that chasing up CNCF again and clarifying our usage is worth doing. I did ask on the issue we already had but haven't yet had a reply from the CNCF member that closed it. (I also don't think I agree with Fabrizio that Packer isn't a runtime dependency for us. The image we produce isn't "image-builder" the process of building is image-builder so that is the runtime in my opinion - but I'm not a lawyer)

Yeah agree it'd be good to have Flatcar on the call at some point. I suspect that @Thilo Fromm might be willing to chat with us as we had a really good chat about it at KubeCon.

If using sysext then it should be possible to use this rather than image-builder to produce the images -

bootc only supporting Fedora/CentOS is too limited right now in my opinion.

/cc @Drew Hudson-Viles @mboersma FYI 🙂

👍 Drew Hudson-Viles, mboersma, Lennart Jern, jawnsy

👍:skin_tone_2: Anurag

Sriraman Srinivasan

2024-06-06 05:45:56

My couple of cents:

I also don't think I agree with Fabrizio that Packer isn't a runtime dependency for us. The image we produce isn't "image-builder" the process of building is image-builder so that is the runtime in my opinion - but I'm not a lawyer

This is my understanding. Please correct me if I am wrong here. The output of the whole process (image-building) are artifacts which are later consumed by CAPX to bootstrap K8s cluster. We use packer to bootstrap VMs in providers and configure further changes and then produce the final artifacts (in case of vSphere OVA, AWS AMI, etc). The use of packer here is only for building the artifacts (or the final desired output) and not while actual cluster creation(this is what I assume is actual runtime - cluster creation). So not sure why calling packer a build time dependency or usage for producing the images is improper.
Again please excuse/correct me if I am wrong here.

Regarding the other points +1. And sorry wanted to attend the SIG call but was out sick, so could not make it.

👍 Lennart Jern

mboersma

2024-06-06 16:39:19

The use of packer here is only for building the artifacts

I agree with this, and my reading of the BUSL license is that's all they care about as well. Hopefully we can get the CNCF to agree.

Thilo Fromm

2024-06-07 09:15:24

OH NO I missed this - was on an off-site all week. I am absolutely available for chatting on CAPI sysexts! @tormath1 is also a good peer for this discussion.

💙 Marcus Noble, tormath1, Anurag, Sriraman Srinivasan

:_thank_you_: Anurag, mboersma

Thilo Fromm

2024-06-07 09:16:49

From all the CAPI providers I guess the OpenStack folks (CAPO) are most experienced with using sysexts at this point. They use Flatcar + Kubernetes sysext a lot for CI / Testing etc. and they really like it.

Mark Kamsika

2024-06-04 10:55:22

@Mark Kamsika has joined the channel

rajas

2024-06-05 06:30:25

@rajas has joined the channel

Anurag

2024-06-07 20:02:46

Hello, have discussed using instead of pip3 as of now in the image-builder repository? Would you be open to it in the current state of the project?

context: I develop mostly from a BM server and I think this would be faster so just initiating a discussion about the same.

mboersma

2024-06-12 19:31:08

That would be interesting to see. uv does seem like a superior tool.
It's mostly a matter of supporting multiple distros cleanly (and Windows). Ideally we stick with "in-the-box" tools packaged by the distro vendor.
We haven't had many actual resolver problems with pip recently, but it's been tricky to install extra packages now that distros want you to use --break-system-packages and --user doesn't always work. Maybe uv would be more flexible here?

Karine Santos

2024-06-11 21:00:22

Hello everyone!
Is it possible to update the kernel version with image builder?

mboersma

2024-06-12 19:25:43

Not directly. image-builder starts from a source image that includes the kernel and all the basic tools, so for the kernel you'll get whatever comes with Ubuntu 24.04 (for example) as packaged for your cloud provider.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-06-13 14:43:02

Image-builder v0.1.30 is now available:
Thanks to all contributors! 🎉

🚀 mboersma, Sriraman Srinivasan, rajas, Anurag

eSean

2024-06-13 15:00:51

@eSean has joined the channel

Jonathan Rosser

2024-06-17 13:23:21

could i get some pointers on the right way to fix the broken azure tests? I can make the whole grub update be conditional on update-grub being installed?

GitHub

Add Ubuntu 24.02 for QEMU by jrosser · Pull Request #1491 · kubernetes-sigs/image-builder

Change description Adds qemu image build for ubuntu 24.04 Related issues Towards: #1406

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1491#issuecomment-2172859153

Slackbot

2024-06-17 15:30:19

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-06-17 15:33:52

As the agenda is currently empty and at least 2 of the maintainers are unable to make it today we're going to skip the office hours this week.

✅ Anurag

Kevin Breit

2024-06-19 02:37:44

@Kevin Breit has left the channel

mloza

2024-06-21 21:22:46

@mloza has joined the channel

Jasper

2024-06-26 14:09:41

@Jasper has left the channel

dalees

2024-06-27 06:08:30

@dalees has joined the channel

bharath2438

2024-07-01 06:07:19

@bharath2438 has joined the channel

Slackbot

2024-07-01 15:30:29

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2024-07-01 16:05:30

We don't have any topics in the agenda currently, so let's skip office hours this week. Please reach out on the Slack channel if you have any questions or need support with image-builder.

Justinas B

2024-07-08 09:22:03

@Justinas B has joined the channel

Justinas B

2024-07-08 10:06:24

Hey! I am facing an issue when building an image - the VM gets stuck on language selection screen and it seems cloud-init scripts are not being executed. Any ideas how to fix that?

Jeremi Piotrowski

2024-07-08 10:27:56

export DEBIAN_FRONTEND=noninteractive

Jeremi Piotrowski

2024-07-08 10:28:05

before running apt-get steps

Justinas B

2024-07-08 11:02:03

I dont see apt-get being invoked anywhere on proxmox build ()

GitHub

image-builder/images/capi/packer/proxmox/packer.json at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/proxmox/packer.json#L60

Justinas B

2024-07-08 11:02:15

Right after below i get language selection screen

SCR-20240708-lrjw-2.png

Justinas B

2024-07-08 11:02:26

any ideas where i should put that export DEBIAN_FRONTEND=noninteractive?

Justinas B

2024-07-08 11:04:50

In logs i see following error "GET /api2/json/nodes/pve-01/qemu/118/agent/network-get-interfaces HTTP/1.1" 500 13, which i assume is caused by qemu agent not running on machine

Justinas B

2024-07-08 13:02:17

Could someone please confirm that image-builder works only if there is a DHCP server to assign IP addresses automatically in proxmox case? It seems that in my case build process gets stuck due to connectivity issues as VM never gets IP address assigned

Nicolò Ciraci

2024-07-08 13:24:45

Hi! Yes, we use it in the same scenario. No issues at all with a simple image with basic Ubuntu. Given the fact that this images should be used primary with CAPI I assume DHCP is always needed. Otherwise you would need to use CloudInit to spin up VMs with already assigned IPs.

Justinas B

2024-07-08 14:19:46

ok, thanks @Nicolò Ciraci! I have spinned up new DHCP server and i see it offered IP address to ubuntu-server

image.png

Justinas B

2024-07-08 14:20:29

but my ubuntu server seems to be stuck here

image.png

Justinas B

2024-07-08 14:21:14

sorry for dumb question - but any ideas why this could be happening and how to debug that?

Justinas B

2024-07-08 14:21:24

packer logs shows below:

2024/07/08 16:15:05 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:05 [INFO] Waiting 5s
2024/07/08 16:15:11 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:11 [INFO] Waiting 5s
2024/07/08 16:15:16 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:16 [INFO] Waiting 5s
2024/07/08 16:15:24 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:24 [DEBUG] Unable to get address during connection step: 500 Internal Server Error
2024/07/08 16:15:24 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:24 [INFO] Waiting for SSH, up to timeout: 2h0m0s
==> proxmox-iso.ubuntu-2204: Waiting for SSH to become available...
2024/07/08 16:15:27 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:27 [DEBUG] Error getting SSH address: 500 Internal Server Error
2024/07/08 16:15:35 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:35 [DEBUG] Error getting SSH address: 500 Internal Server Error
2024/07/08 16:15:43 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:43 [DEBUG] Error getting SSH address: 500 Internal Server Error

Nicolò Ciraci

2024-07-08 14:34:03

The packer errors are caused by the fact that your instance of Ubuntu is stuck, but I don’t see any clear issue in the logs. Are DNS reachable?

Justinas B

2024-07-08 14:45:43

yeah, if I restart machine and and login through serial console, i am able to manually deploy qemu-guest-agent

Justinas B

2024-07-08 14:46:03

but it needs restart, as it is literraly stuck

Justinas B

2024-07-08 14:46:10

i do not see any network activity happening on it

Nicolò Ciraci

2024-07-08 15:03:49

The image-builder stack has some additional flag by which you can install additional package. Something like this PACKERFLAGS="--var 'extradebs=\"qemu-guest-agent\"'" make build-qemu-ubuntu-2204-crio ; you’ll need to adjust the command based on you linux flavour.

Justinas B

2024-07-10 07:02:18

OK, so it seems the issue was that my laptop where i ran packer was not accessible from the temporary VM

Justinas B

2024-07-10 07:02:30

i have moved the build server to proxmox itself and it worked fine

chrischdi

2024-07-11 16:58:39

@chrischdi has joined the channel

chrischdi

2024-07-11 16:59:33

Hey folks, I opened an issue regarding the mandatory pull-ova-all due to the upcoming changes by test-infra / to the prow CI, PTAL:

GitHub

Keep mandatory presubmit ci working after 1st of august · Issue #1508 · kubernetes-sigs/image-builder

Hey folks, the presubmit prowjob pull-ova-all is backed by infrastructure which is not owned by the community. Starting by the 1st of August this job will not work anymore. (see this mail for more ...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1508

Slackbot

2024-07-15 15:30:26

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-07-15 15:33:01

Agenda is currently empty. Does anyone have any topics they'd like to discuss or shall we skip?

mboersma

2024-07-15 15:39:38

Nothing in particular on my end, happy to skip it or to carry on if someone has a discussion topic.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-07-15 16:03:45

ok, unless someone shouts up in the next 30 min I'm going to skip. Could do with a rest after work to be honest 🙂

❤️ mboersma

Drew Hudson-Viles

2024-07-15 16:11:02

What about the OVA removal that's happening? Worth chatting abotu that or shall we put it off until next time? I've got nothing to add just wasn't sure if a discussion was needed 😄

Drew Hudson-Viles

2024-07-15 16:11:32

But yeah. I'm running on 3 hours sleep thanks to poorly child so I would appreciate not going on YouTube this week 😄

mboersma

2024-07-15 16:16:57

I don't have much insight into the OVA changes, but it would be good to hear from someone more involved. If we're not up for it this week, maybe we can recruit someone to summarize next time.

Drew Hudson-Viles

2024-07-15 16:17:33

yeah that sounds like a good plan

Marcus Noble (k8s@marcusnoble.co.uk)

2024-07-15 16:18:10

Yeah agreed

Karine Santos

2024-07-17 13:55:49

Hello Everyone!
I'm trying to upload an image from 1.29 and 1.30 of k8s, and I saw that the cni-plugin was upgraded from that version (before I used v1.2.0 of the cni-plugin, and from version 1.29 and 1.30 of k8s , cni-plugin updated to v1.3.0 and v1.4.0), but I get the following error when trying to update cni-plugin configuration in packer?

Screenshot from 2024-07-17 09-27-24.png

Karine Santos

2024-07-17 13:56:24

Before I configured it this way. Here are the changes I made:

Screenshot from 2024-07-17 09-28-27.png

Karine Santos

2024-07-17 13:56:54

The error I get:

openstack: fatal: [default]: FAILED! => {"cache_update_time": 1721186207, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubelet=1.29.6-1.1' 'kubeadm=1.29.6-1.1' 'kubectl=1.29.6-1.1' 'kubernetes-cni='' failed: E: Version '' for 'kubernetes-cni' was not found\n", "rc": 100, "stderr": "E: Version '' for 'kubernetes-cni' was not found\n", "stderr_lines": ["E: Version '' for 'kubernetes-cni' was not found"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nPackage kubernetes-cni is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package kubernetes-cni is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}
4116    openstack:
4117    openstack: PLAY RECAP **
4118    openstack: default                    : ok=44   changed=34   unreachable=0    failed=1    skipped=190  rescued=0    ignored=0
4119    openstack:

Screenshot from 2024-07-17 09-29-54.png

Anurag

2024-07-17 17:03:30

I think you'll have to set 1.4.0 for kubernetescnideb_version at least given you're using apt.

Is there a reason to set to it to null? Looking at the config I think that should be 1.4.0

Sriraman Srinivasan

2024-07-18 13:27:43

Interesting discussion -->

GitHub

Setting a standard for VM images to be used by CAPI · Issue #10900 · kubernetes-sigs/cluster-api

What would you like to be added (User Story)? As a developer I would like a list of components and standards that make a VM image to be used by CAPI Detailed Description Currently, cluster-api uses...

Original URL: https://github.com/kubernetes-sigs/cluster-api/issues/10900

Junfei Zhang

2024-07-23 00:50:37

@Junfei Zhang has joined the channel

Junfei Zhang

2024-07-23 01:10:13

Hey everyone! quick question is this PR still functional ? In other word, is building arm64 image on mac m1 supported by image-builder?

GitHub

AARCH64 / ARM64 CAPI image build for Ubuntu 22.04 by ader1990 · Pull Request #1142 · kubernetes-sigs/image-builder

What this PR does / why we need it: Added example on how to build an ARM64 CAPI image using QEMU. How to run: # under root apt-get install qemu-system-arm qemu-efi qemu-kvm libvirt-daemon-system -...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1142

:think3d: Phil H

Vladu

2024-07-23 09:01:58

Hello, I am the creator of the PR. This PR was done aa year ago and will probably not work anymore because of the now inexistent links to the k8s binaries. Still, with some changes to the urls, it might work on a native arm64 Linux box. If I remember correctly, I know Mac M1 does not actually support kvm qemu-arm64? if that is the case, you need to use a Linux arm64 box to try out the PR.

Vladu

2024-07-23 09:03:13

We can continue this discussion on the PR github thread, to let me know if you are constrained to running this on Mac M1. Fortunately, I also own a Mac M1 and I can check it out too.

Marius Oprin

2024-07-23 04:19:58

@Marius Oprin has joined the channel

Jay J

2024-07-23 05:56:21

@Jay J has joined the channel

Vladu

2024-07-23 08:59:19

@Vladu has joined the channel

feitnomore

2024-07-25 00:32:00

does the images for kubevirt/qemu support EFI/Secure Boot? I've tried to boot a few of them and failed

feitnomore

2024-07-25 00:35:18

follow up question: is it possible to create an image that supports both, bios and efi, as well as secureboot? I see cloud images like ubuntu support that

Jeremy Maes

2024-07-25 11:10:52

@Jeremy Maes has joined the channel

Tim Cheng

2024-07-26 05:06:45

@Tim Cheng has joined the channel

Slackbot

2024-07-29 15:30:22

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Sriraman Srinivasan

2024-07-30 11:30:25

@Marcus Noble /@mboersma Need help with merging PR. This is regarding Ubuntu-24.04 support for vSphere environment. All the CI jobs are successful. It would be great if we can get this in before end of this month (next two days) - disabling of pre-submit CI for all OVA issues
cc: @chrischdi @rajas

GitHub

Add Ubuntu 24.04 LTS for VMware vSphere by sriramandev · Pull Request #1459 · kubernetes-sigs/image-builder

Change description Adds support for Ubuntu 24.04 LTS for VMware vSphere Related issues Refs: #1406 See: #1003 Additional context ubuntu 24.04 now uses Python 3.12. With Python 3.10 distutils pa...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1459

Christian Schlotter (https://kubernetes.slack.com/team/UBVL9Q4TB)

Hey folks, I opened an issue regarding the mandatory <code>pull-ova-all</code> due to the upcoming changes by test-infra / to the prow CI, PTAL: <a href="https://github.com/kubernetes-sigs/image-builder/issues/1508">https://github.com/kubernetes-sigs/image-builder/issues/1508</a>

Original URL: https://kubernetes.slack.com/archives/C01E0Q35A8J/p1720713573345199

:this: rajas

🎉 chrischdi, rajas

:lgtm: mboersma

rajas

2024-07-30 11:52:07

Yay! Finally the CI is happy! 🎉

Sriraman Srinivasan

2024-07-31 05:52:24

Thanks @mboersma @Drew Hudson-Viles for the approval and merge in time...

:gr_captain_salute: Drew Hudson-Viles

:thank_you: mboersma

bavarianbidi

2024-07-30 14:28:38

Hey folks,
question-block regarding the build process how it's implemented in image-builder.

not really related to image-builder project itself - but i see a large user base here --> more distributed knowledge 🤞

What's the main reason why image-builder doesn't build a generic raw and convert it later into a qcow2 (openstack-square ), an ami (aws ) or vhdx (azuretent )?

just asking because:

i had to build a generic OS image which has to be used on dedicated cloud platforms

and TIL thatqemu-img is able to convert images

Does it brings any downside for using a converted image, compared to a dedicated build artifact for e.g. Azure)? 🧵

bavarianbidi

2024-07-30 14:29:40

Or is it for historical reasons, that no generic build was implemented as in the beginning of image-builderit was easier to build on the target cloud-providers directly?

chrischdi

2024-07-30 16:04:34

I think one point is e.g.: For azure you may want to have different config for e.g. cloud-init or tools installed e.g. the azure cli, compared to when running on openstack.

There are some ansible tasks which are run depending on the target which gets built.

bavarianbidi

2024-07-31 06:18:15

thank you Christian, yes that's a valid point.
And if i take a look at the provider specific ansible tasks, they seems to be valid for an image-builder perspective (like you said, having azure-cli in there or the ssm agent for AWS.

but if you do not really need these, do you know any limitiation when it came to conversion via qemu-img convert

chrischdi

2024-07-31 07:37:34

Nope, not really. If the image still has everything you need, I don’t see a reason why it should not work, except maybe hypervisor specific things at the end, but if it runs it runs 🙂

:thx_thanks: bavarianbidi

chrischdi

2024-07-31 07:38:00

Note: I’m not very familiar with aws and azure image building.

:this_is_fine_fire: bavarianbidi

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-01 10:24:13

🤔 I wonder if this is a good approach for us to try out switching to using systemd sysext. Instead of trying to update all existing targets we create a new one specifically for sysext that builds a raw image with optional conversion at the end for each cloud provider. That convert stage could also possibly layer on provider-specific sysext packages like azure-cli.

chrischdi

2024-08-01 10:50:40

That would be way more efficient ack

tormath1

2024-08-01 10:54:49

Giving my 2 cents, this is how Flatcar is actually built: there is a generic image build then it's converted to cloud providers VM images with specific bits (image format, OEM tools provided as sysext, etc.)

👍 Marcus Noble

:thx_thanks: bavarianbidi

mboersma

2024-07-31 00:06:42

Image-builder v0.1.31 is now available: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.31
Thanks to all contributors! 🎉

:awesome_go: chrischdi, Anurag

:emoji_k8s_loft: Drew Hudson-Viles, Anurag, jsturtevant

hbarel

2024-08-01 13:27:17

@hbarel has joined the channel

Erkan Erol

2024-08-02 17:08:52

@Erkan Erol has left the channel

quba

2024-08-05 16:32:21

@quba has left the channel

bentheelder

2024-08-06 23:53:11

@bentheelder has joined the channel

bentheelder

2024-08-06 23:53:53

PTAL, there's a bug in acquiring GCP projects that is causing leaked CI projects

GitHub

use dirty state when releasing boskos projects by BenTheElder · Pull Request #1534 · kubernetes-sigs/image-builder

Change description Switches from &quot;used&quot; to &quot;dirty&quot; for boskosctl project release Related issues Fixes # Additional context I&#39;m pretty sure this usage explains why we ha...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1534

:done2: jsturtevant

dims

2024-08-07 01:34:23

@jsturtevant around? PTAL if so!

dims

2024-08-07 01:47:59

thanks a ton @jsturtevant

💯 bentheelder

❤️ bentheelder

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-09 10:08:09

Has anyone been able to successfully build the latest 3975.2.0 release of flatcar Flatcar for CAPV using make build-node-ova-vsphere-flatcar?
I'm getting stuck at the "Waiting for IP..." stage but currently don't have access to vcenter to debug the issue 😞
(The exact same image-builder setup works fine with Flatcar 3815.2.5 so I suspect something has changed in the new release that we need to handle)

chrischdi

2024-08-09 10:27:49

what could help for debugging (also in future) is try to get a screenshot of the vm to the artifacts using govc vm.console -capture screen.png my-vm

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-09 10:28:26

I don't currently have access to vcenter. Needing another team to take a look for me.

chrischdi

2024-08-09 10:28:50

I’m planning to build an image next week for CAPV with v1.31 when its released.

chrischdi

2024-08-09 10:29:31

I don’t currently have access to vcenter. Needing another team to take a look for me.

I was thinking of having this command somewhere in the image-builder pipeline for failures 🙂 to always get this in case of an error, also later when ci on vsphere is back.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-09 10:29:45

Oh interesting idea!

chrischdi

2024-08-09 10:30:49

or at least being able to toglge it on manually on a PR or so if it can’t be always on 🙂

:nod: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-09 10:31:36

But thinking about it, in my CI pipeline I don't actually have any way currently to save artifacts like that so would need a bit of a rewrite on my end anyway

chrischdi

2024-08-09 10:32:05

ah ok 😞 fair point

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-09 10:32:29

Still, extra debugging of failures is always helpful

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-09 16:09:34

Update: Looks like the latest Flatcar stable release has a considerable slower boot time (by 1m30s) that is causing problems with the default bootwait time we have set in image-builder not being long enough.
To work around this issue until Flatcar has worked out what is the cause of the slowdown you can set the bootwait Packer variable to something like 120s in your user provided vars.

:thx_thanks: chrischdi

chrischdi

2024-08-13 19:22:27

120s was not enough over here, did use 180s instead (~145s would have been good enough).

Most time is for:

image.png

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 19:24:22

Yeah, exactly that. I found 120s worked for me ( and I have a PR to set that) but it doesn’t hurt to have a little more possibly.
I did also have trouble with the boot command itself failing too.

👍 chrischdi

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 19:24:37

https://github.com/kubernetes-sigs/image-builder/pull/1540

GitHub

Better handling of Flatcar iso boot command by AverageMarcus · Pull Request #1540 · kubernetes-sigs/image-builder

Change description Increases the boot_wait for Flatcar for OVA and updates the boot commands to be more resilient to the delay in some of the commands running (that can prevent the reboot command ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1540

chrischdi

2024-08-13 19:25:46

Let’s see if that commands worked for me 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 19:45:06

I don’t understand why it’s failing in the latest flatcar release though

chrischdi

2024-08-13 19:46:01

which one?

With your PR I’m now at the ansible stuff (stable channel / 3975.2.0)

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 19:52:34

I mean I don’t understand what changed in the latest flatcar that has caused what was there before in image builder to fail

:nod: chrischdi

chrischdi

2024-08-13 19:52:56

If I find time I may take a look the next days.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 20:00:12

There’s some updates from the flatcar team in that issue I opened

Slackbot

2024-08-12 15:30:24

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-12 15:39:46

☝️ Agenda is currently empty. Anyone got anything they'd like to discuss or shall we skip?

mboersma

2024-08-12 16:05:13

I don't have anything in particular, happy to skip unless someone adds something to the agenda.

👍 Marcus Noble

Drew Hudson-Viles

2024-08-12 16:33:34

Sorry for the late reply, yes I'm happy to skip. I'm actually helping my brother move today anyway.

Dhilip Shankaranarayanan

2024-08-12 16:00:41

@Dhilip Shankaranarayanan has joined the channel

bentheelder

2024-08-12 16:52:20

@bentheelder has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 10:19:56

Can someone please help me sanity check if an assumption I previously had about AMI builds is actually incorrect?

In my image-builder pipelines I'm setting the FLATCARVERSION env var to specify what Flatcar image to base my build on. This works as expected on Azure thanks to setting the distributionversion in the packer vars but I don't think this env var is used when building AWS AMIs. 😞

For AMIs we use the following AMI filter:

Flatcar{{env FLATCAR&#95;CHANNEL}}

which just seems to take into consideration the flatcar channel.

For reference, a Flatcar AMI name looks something like this: Flatcar-stable-3975.2.0-hvm

Unless I'm mistaken, this means that when building AMIs image-builder will ALWAYS use the latest for a given Flatcar channel and there's no way to override that.

Is that correct? (It shouldn't be but pretty sure we've missed this)

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 10:23:19

I'm pretty sure updating the filter to the following would fix things:

Flatcar{{env FLATCAR&#95;CHANNEL}}{{env FLATCAR&#95;VERSION}}**

Jeremi Piotrowski

2024-08-13 10:29:28

what you're sayingmakes sense to me

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 10:30:29

😞 Thanks for confirming. Guess I'll get a fix PR up shortly then

👍 Jeremi Piotrowski

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 10:37:47

GitHub

Allow specifying Flatcar version for base AMI by AverageMarcus · Pull Request #1538 · kubernetes-sigs/image-builder

Change description This PR updates the ami_filter_name used for Flatcar when building AMIs so that it supports the FLATCAR_VERSION environment variable being set like other providers do. Related i...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1538

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-13 15:04:23

Image-builder v0.1.32 is now available:
Thanks to all contributors! 🎉

GitHub

Release v0.1.32 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.32">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.32</a> What's Changed Update docs for image-builder v0.1.31 release...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.32

🎉 Drew Hudson-Viles, Abhay Krishna Arunachalam, Anurag, Sriraman Srinivasan

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-15 09:48:51

Image-builder v0.1.33 is now available:
Note: This is a small fix release to resolve the above issue with the latest Flatcar ISO release. There's not rush to upgrade to this if you don't use the Flatcar ISO. 🙂

GitHub

Release v0.1.33 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.33">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.33</a> What's Changed Updated docs for v0.1.32 by @AverageMarcus in...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.33

:partyk8s: Drew Hudson-Viles, chrischdi, Anurag, jsturtevant, Sriraman Srinivasan

Rende Luitjes

2024-08-19 10:54:20

@Rende Luitjes has joined the channel

Clark Zinzow

2024-08-21 07:42:02

@Clark Zinzow has joined the channel

Frans

2024-08-26 02:28:59

@Frans has joined the channel

Yongxiang Gao

2024-08-26 10:38:31

In ubuntu 22.04, we can find file /etc/default/grub, and there is line like this:

GRUB_CMDLINE_LINUX=" apparmor=1 security=apparmor"

How is it generated?
If I want to add something like "intel_iommu=on modprobe.blacklist=nouveau", how to do so within image-builder source code?

Sriraman Srinivasan

2024-08-26 11:51:20

If you have added the gpu role as part of your nodecustomrolespre and enabled the blocknouveau_loading variable (set to true ), nouveau driver will be blocklisted and kernel will not load the module. This disabling is done as part of gpu role.

GitHub

image-builder/images/capi/ansible/roles/gpu/tasks/main.yml at 813df6b2ea942e2e23bd6ce907a9846360b49d48 · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/813df6b2ea942e2e23bd6ce907a9846360b49d48/images/capi/ansible/roles/gpu/tasks/main.yml#L16

image.png

:ty: Yongxiang Gao

Yongxiang Gao

2024-08-26 16:52:21

I will try it out.
BTW, how about "inteliommu=on"?
Can this trick be use together with yours?
- name: Enable IOMMU
ansible.builtin.lineinfile:
path: /etc/default/grub
regexp: '^GRUBCMDLINELINUXDEFAULT="((?:(?!inteliommu=on).)**?)"$'
line: 'GRUBCMDLINELINUXDEFAULT="\1 intel_iommu=on"'
backup: true
backrefs: true
notify: update-grub

Sriraman Srinivasan

2024-08-27 07:02:32

Similar approach has been used for RedHat linux setting huge pages. You can do this in a custom role and invoke the role.

Also if not mistaken you can create a config file with the values in /etc/default/grub.d folder similar to the way we configure non-graphical consoles for debian systems.

Another way could be to update grub in boot config(cloud-init, etc) depending upon your OS. ---> This is something I have not tried, but could work(I may be wrong here)

GitHub

image-builder/images/capi/ansible/roles/node/tasks/main.yml at 813df6b2ea942e2e23bd6ce907a9846360b49d48 · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/813df6b2ea942e2e23bd6ce907a9846360b49d48/images/capi/ansible/roles/node/tasks/main.yml#L117

GitHub

image-builder/images/capi/ansible/roles/sysprep/tasks/debian.yml at 813df6b2ea942e2e23bd6ce907a9846360b49d48 · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/813df6b2ea942e2e23bd6ce907a9846360b49d48/images/capi/ansible/roles/sysprep/tasks/debian.yml#L107

Yongxiang Gao

2024-08-27 10:48:26

Thanks, yes, I'm able to add it as follows:
- name: Enable IOMMU
ansible.builtin.lineinfile:
path: /etc/default/grub
backup: true
backrefs: true
regexp: ^GRUBCMDLINELINUX="((?:(?!inteliommu=on).)**?)"$
line: GRUBCMDLINELINUX="\1 inteliommu=on modprobe.blacklist=nouveau"
when: ansible_distribution == "Ubuntu"

Sriraman Srinivasan

2024-08-27 12:03:06

The above one will blacklist/block module from initially loading the module. However, if some other driver or something else tries to load it, it will still get loaded.

Yongxiang Gao

2024-10-21 06:30:21

What's the better way to add?

Yongxiang Gao

2024-10-21 09:43:59

Someone has luck to create EKSA vSphere cluster with ubuntu 22.04 at all?

Sriraman Srinivasan

2024-10-21 10:25:46

What's the better way to add?

Depends on whether you are likely to have a scenario where you have someother entity/driver trying to load nouveau kernel package. If yes and you wish to always make sure it does not load the package, you could add install nouveau /bin/false along with modprobe blocklisting. If thats not the case or does not matter if later package is loaded, what you have added should work.

:ty: Yongxiang Gao

Slackbot

2024-08-26 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2024-08-26 15:47:36

I'm not going to be able to attend this morning, but we probably have some things to discuss. If we don't have anything on the agenda and don't meet, I'll be here most of the day to discuss whatever on Slack.

Sriraman Srinivasan

2024-08-28 06:49:59

@mboersma /@Marcus Noble Can we close this issue since there has been no response?

GitHub

Add "http_port_min" and "http_port_max" configuration for cloud-init · Issue #1423 · kubernetes-sigs/image-builder

This issue requests the addition of two new variables, http_port_min and http_port_max, to all packer.json files in the project. These variables will allow users to specify a port range for the HTT...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1423

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-28 07:07:08

Yeah I think so. Can always be reopened if needed

🙏 Sriraman Srinivasan

Scott Lowe

2024-08-28 23:45:25

@Scott Lowe has joined the channel

Scott Lowe

2024-08-28 23:51:49

Hey folks, I'm trying to use Image Builder to build an AWS AMI that will be used for an "offline" (no Internet access) K8s cluster. I need to load some additional container images into the AMI. To do that, I made the following changes to the files in my cloned copy of the Image Builder repo:

In images/capi/ansible/roles/loadadditionalcomponents/defaults/main.yml, I set additionalregistryimages to "true".

I provided a comma-delimited list of registry images in additionalregistryimages_list. The registry images are currently specified as "quay.io/path/image:tag".

Are there any other changes I need to make? Anything I'm missing from what you can tell?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 07:03:44

I vaguely recall some discussion about airgapped environments a few months ago. I can't remember the details though unfortunately but I you might be able to search this channels history.

👍:skin_tone_2: Scott Lowe

Scott Lowe

2024-08-29 15:07:08

Hmm, looks like the airgap discussion was about image-builder itself, not about building images for airgapped environments. 😕

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 15:10:36

Ah sorry. 😞

Scott Lowe

2024-08-29 15:11:42

No worries! I appreciate the response. Ignoring the airgap portion for a moment, do the changes I mention above sound correct for preloading additional container images into an AMI?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 15:12:15

I think so. But to be honest I'm not entirely sure as never used it myself.

Scott Lowe

2024-08-29 15:13:30

Fair enough. Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 15:14:16

Please report back how you get on! I'd be keen to hear if there was anything else.

Scott Lowe

2024-08-29 15:16:49

I will almost certainly write a blog post with details once I figure this out! ✍️

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 15:17:07

Perfect! 😄

Scott Lowe

2024-08-29 18:13:47

Here's the solution. All changes need to go into images/capi/packer/config/additionalcomponents.json. (No changes need to be made to the Ansible roles.) There are three changes to make:

Set loadadditionalcomponents to "true".

Set additionalregistryimages to "true".

Specify a comma-delimited list of fully-qualified image names (i.e., "quay.io/path/image:tag") in additionalregistryimageslist.

The resulting AMI will have all the "base" container images as well as the additional container images you specified.

:parrotk8s: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 07:00:57

Cross-posting this in case anyone else here has tried building the latest Flatcar on Azure yet. (Maybe @mboersma 🙏 )

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)

Have the permissions of <code>/usr/share</code> changed in the latest Azure release of Flatcar (<code>3975.2.0</code>)? I'm now getting the following error when running <a href="https://github.com/kubernetes-sigs/image-builder">image-builder</a> against it with no changes from the previous stable release. <code>azure-arm.sig-flatcar-gen2: TASK [providers : Configure PTP] ********************************************************************************************** azure-arm.sig-flatcar-gen2: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 18] Invalid cross-device link: b'/tmp/.ansible/ansible-tmp-1724908280.8035567-1467-271096777552195/tmplopt3bfo' -&gt; b'/usr/share/oem-azure/chrony.conf' azure-arm.sig-flatcar-gen2: fatal: [default]: FAILED! =&gt; {"changed": false, "msg": "The destination directory (/usr/share/oem-azure) is not writable by the current user. Error was: [Errno 30] Read-only file system: b'/usr/share/oem-azure/.ansible_tmpyspstzmwchrony.conf'"}</code> (I'll be honest, I don't fully understand why Packer is using that directory as <a href="https://github.com/kubernetes-sigs/image-builder/blob/71c69132caced034f4f58cf7516de7dd36cf7ec7/images/capi/ansible/roles/providers/tasks/azure.yml#L18-L22">this is the task it is running</a> which specifies <code>/etc/chrony/chrony.conf</code> - is there a symlink here or something?)

Original URL: https://kubernetes.slack.com/archives/C03GQ8B5XNJ/p1724911221970159

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 11:11:46

github

GitHub

Fix timesyncd support in latest Flatcar release for Azure by AverageMarcus · Pull Request #1551 · kubernetes-sigs/image-builder

Change description The latest stable release of Flatcar now includes default support for ntp on Azure which is causing the following issue when trying to build with image-builder: azure-arm.sig...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1551

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 11:56:10

☝️ @Drew Hudson-Viles if you're able to would you mind looking at this too?

👍 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 08:07:30

😩 To make things worse, ubuntu:latest now points to Ubuntu 24.04 which apparently doesn't have a qemu package available to install. Today is not going well for me 😅
Do y'all think we should pin the base ubuntu image to the previous (22.04)?

Drew Hudson-Viles

2024-08-29 08:57:40

I could be wrong here - but I think you have to install qemu-system since 23.xx

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 10:10:35

Yup, figured that out. 🙂

Then hit an issue where 24.04 now has the ubuntu user as 1000 and imagebuilder gets created as 1001 which broke all my permissions 😩

I'm not having a great morning so far.

Drew Hudson-Viles

2024-08-29 10:12:21

You need a weekend 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 10:18:18

I need a holiday I think 😅 This week has been A LOT

☝️ Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 10:19:14

But yeah, we might want to think about pinning the version as anyone building on top might experience similar issues. I suspect we should at least have it upgraded in a new release (maybe we pin to 24.04 and have that as a new release?)

👍 Sriraman Srinivasan

Drew Hudson-Viles

2024-08-29 10:23:26

yeah that sounds sensible to me

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 10:24:11

I'll try and get a PR up shortly. Currently trying to fix the above flatcar problem first as it's causing us problems 😞

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 10:38:36

Oh fun! I've also just learnt that Goss failing doesn't actually result in the make command failing 🤦‍♂️

:shake_fist: Drew Hudson-Viles, nikparasyr

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 11:17:58

github

GitHub

Pin base container image to Ubuntu 24.04 by AverageMarcus · Pull Request #1552 · kubernetes-sigs/image-builder

Change description The ubuntu:latest tag has now been updated to point to 24.04 which caused some surprises when I was trying to build a new image built on top of image-builder as some packages ha...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1552

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 13:02:33

🤦‍♂️ The PR also needs the /lgtm label. (No idea why when it has the approve 🤷 )

Sriraman Srinivasan

2024-08-30 05:36:17

Oh fun! I've also just learnt that Goss failing doesn't actually result in the make command failing

Realized that couple of days back when working on our systems using IB... goss validations failed but build succeeded.... wanted to raise a issue but forgot...

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 06:21:09

🤦‍♂️ i also forgot with everything else I was dealing with yesterday. I’ll create one today!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 08:00:29

🤦‍♂️ And I've just noticed thanks to writing up this issue that my CAPA builds have Goss failing

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 08:08:33

github

GitHub

Goss failure doesn't result in a failure exit code · Issue #1554 · kubernetes-sigs/image-builder

Environment Make target: make build-ami-flatcar (but applies to other targets) Run using container image? (Y/N): Y Environment vars: Vars file: { "ssh_clear_authorized_keys": "true", "kubernetes_de...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1554

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 08:18:37

GitHub

Goss failure doesn't result in a failure exit code · Issue #1554 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1554#issuecomment-2320311304

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 08:39:24

Will need to fix this first before handling the above:

GitHub

Goss validate failing for `Service: systemd-timesyncd: running` · Issue #1555 · kubernetes-sigs/image-builder

Environment Make target: make build-ami-flatcar Run using container image? (Y/N): Y Environment vars: Vars file: { "ssh_clear_authorized_keys": "true", "kubernetes_deb_version": "1.30.4-00", "kuber...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1555

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-29 17:07:00

Image-builder v0.1.34 is now available:
Thanks to all contributors! 🎉

Please note that this release include the above mentioned bumping of the base Ubuntu image from 22.04 -> 24.04.

GitHub

Release v0.1.34 · kubernetes-sigs/image-builder

Image Builder v0.1.34

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.34

Brennen Murray

2024-08-29 22:54:55

@Brennen Murray has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 08:09:25

☝️ I don't really know much about Goss so not sure I can work on fixing it, at least not right now anyway, so if anyone is able to help I'd very much appreciate it! 💙

Edit: A few minutes searching and I think I've figured it out 😆

😄 Drew Hudson-Viles

Drew Hudson-Viles

2024-08-30 09:02:02

You're too ninja fast for me. I log in and you've both found a problem and may have a fix 😄

I don't have an Amazon env to build these in unfortunately either. But I can take a look from the OpenStack perspective if needed still.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 09:03:41

It looks like its Flatcar specific. It's just I had logs saved for my previous AMI builds 🙂 (Edit - the failing goss test is flatcar specific)

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 09:03:53

Anyway, I have a fix PR incoming 😉

👍 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 09:05:48

Fix for the Flatcar failure:
Ensure Goss failure fail the image build:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 09:06:21

(I'm having a MUCH more productive day so far than I did yesterday 😆 )

😆 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 09:10:08

I also feel like I might finally be starting to get a grip on how most of image-builder fits together 🤣

Drew Hudson-Viles

2024-08-30 09:12:37

Yeah I feel like I'm finally there now. The attempt to switch to HCL (which I've put on hold until we know where we stand with packer) made me learn a significant amount about the structure and more importantly the beast of a Makefile.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-08-30 09:14:11

Oh the Makefile is still a "here be dragons" thing for me. I can tweak it a little but I wouldn't feel comfortable making any large changes to it 😅

😄 Drew Hudson-Viles, Sriraman Srinivasan

Gareth

2024-09-01 05:57:07

@Gareth has joined the channel

Karthik-K-N

2024-09-03 07:38:41

@Karthik-K-N has joined the channel

Karthik-K-N

2024-09-03 07:53:51

Hi Team, Trying to understand about previous work or future plan for supporting fedora based image building for CAPI. Could some one please help.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-03 07:55:35

What are you wanting to know exactly?

There is no dedicated support to any OS or provider and we rely on community contributions. Are you looking to help out with Fedora support?

Karthik-K-N

2024-09-03 07:56:41

Yes, I am looking forward to use fedora for creating a CAPI cluster

Karthik-K-N

2024-09-03 07:57:37

I see that currently image-builder does not have support for building fedora images, So looking forward to understand about plan for adding fedora support or how can I add the support

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-03 08:01:41

First place to start would likely be to open an issue on the repo stating what versions of Fedora you're looking for and against what providers.

I know we have CentOS support for AWS at least so that might be a good place to look at what might be needed for adding Fedora.

Karthik-K-N

2024-09-03 08:02:56

Sure, Thanks for the quick help. I will create an issue and possible try to add fedora support as well.

👍 Marcus Noble

Karthik-K-N

2024-09-03 10:00:19

Created issue for reference

GitHub

Add support for building CAPI compatible Fedora CoreOS images · Issue #1559 · kubernetes-sigs/image-builder

Is your feature request related to a problem? Please describe. Describe the solution you'd like I would like to provision CAPI cluster with fedora coreos VM using IBM Cloud Power VS infrastructure ...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1559

:thank_you_icon: Marcus Noble

Karthik-K-N

2024-09-04 05:22:45

@Marcus Noble By any chance do you have any doc or reference or PR on what are the things needed to be done to add a new OS support. Just looking for some reference

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-04 05:31:59

We dont have any docs unfortunately (we prob should 😔 ) but I could likely find an example PR for you. Let me see what I can find…

Karthik-K-N

2024-09-04 05:32:20

sure that will be really helpful.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-04 05:38:51

Here’s a pretty large PR adding a totally new OS distro: https://github.com/kubernetes-sigs/image-builder/pull/1192

And here’s a couple small ones just adding newer versions of existing distros:
https://github.com/kubernetes-sigs/image-builder/pull/1500
https://github.com/kubernetes-sigs/image-builder/pull/1476

Karthik-K-N

2024-09-04 05:39:56

Nice, Thank you I will use this as a reference.

Kepler SysAdmin

2024-09-04 17:40:30

When Define variables for OpenStack build,
What version are available for variable kubernetes_version?
ref -

Drew Hudson-Viles

2024-09-04 17:46:44

It just depends which version of Kubernetes you want. You can find a list on the Kubernetes releases page.

Kepler SysAdmin

2024-09-04 17:47:24

here: ?
What happens if i want an version that is not longer in the list?

Drew Hudson-Viles

2024-09-04 17:48:44

It likely won't be supported as image builder is configured to use the new repos which iirc started at 1.27 or 1.28. Also, they likely aren't maintained any more so it wouldn't get the latest updates or security patches.

Kepler SysAdmin

2024-09-04 17:49:25

What is the list?
This: ?

Kepler SysAdmin

2024-09-04 17:57:37

basically what are the supported version when using the image builder?

Drew Hudson-Viles

2024-09-04 18:20:51

Yeah that's the list. The only ones we can really support are the same as are listed in that link as anything further back is not supported by the Kubernetes community

👍 Kepler SysAdmin

Kepler SysAdmin

2024-09-04 21:34:36

Thank you!

Daz Mac

2024-09-09 05:02:48

@Daz Mac has joined the channel

Slackbot

2024-09-09 15:30:20

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2024-09-09 16:02:14

Anything to discuss this week? I know we have one ongoing item but do we need to catch up today on this?

Drew Hudson-Viles

2024-09-09 16:37:55

Skipping this week due to no attendance

vikram

2024-09-11 05:53:32

@vikram has joined the channel

Anil Hariharan

2024-09-11 06:03:41

@Anil Hariharan has joined the channel

Anil Hariharan

2024-09-11 06:13:44

Hi, we are currently trying to generate ubuntu 22.04 image with kubernetes v1.28.0 but keep running into problems. Can somebody please help?

GitHub

Error in creating ubuntu 22.04 with k8s 1.28.1 image · Issue #1565 · kubernetes-sigs/image-builder

Environment Make target: make build-kubevirt-qemu-ubuntu-2204 **Run using container image? (Y/N):N Environment vars: None Vars file: None What steps did you take and what happened? -git clone http...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1565

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-11 07:05:09

I’m no expert on qemu but this line looks relevant:

2024/09/10 09:37:30 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/09/10 09:37:30 Qemu stderr: Unable to init server: Could not connect: Connection refused

Maybe some sort of permission or connection issue? I haven’t seen this before myself.

Anil Hariharan

2024-09-11 07:32:39

im able to build ubuntu 20.04 with k8s v1.28, but run into error when its ubuntu 22.04 with k8s v1.28

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-11 07:48:38

Oh that’s weird. I’m not aware of anything that might be different unfortunately 😔 Hopefully someone else has some suggestions. I’m currently travelling so not able to test myself currently but @Drew Hudson-Viles might be able to if he’s about this week? 🤞

Drew Hudson-Viles

2024-09-11 08:39:16

sure thing, let me take a look

💙 Marcus Noble

Drew Hudson-Viles

2024-09-11 08:59:56

I've managed to build this myself using the same command. the only difference is I didn't run it with sudo. I get a different error when running with sudo as I don't have the packages available in the PATH. It is definitely worth checking if this is the source of the problem though. But I can confirm the process definitely works.

I've dropped a commend in the issue anyway.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-11 09:00:29

Might be worth trying with the container image too is possible

Drew Hudson-Viles

2024-09-11 09:00:37

Indeed

Anil Hariharan

2024-09-11 13:53:34

even without sudo it gets stuck at qemu: Waiting for SSH to become available...

Drew Hudson-Viles

2024-09-11 13:54:54

I'd recommend launching the vnc and checking what's happening on the VM it's launched in this case. I've ran through this on my side and I can confirm it's 100% working.

The other option would be to try using the Docker image that's supplied to run a build.

Anil Hariharan

2024-09-13 05:36:10

we were able to generate the image now, Thanks!

🙌 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-15 10:03:03

What was the solution in the end?

Anil Hariharan

2024-09-16 05:43:24

something wrong with the dev environment we were working on, tried it from scratch on a new machine and it worked!

mboersma

2024-09-19 18:21:38

Image-builder v0.1.35 is now available: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.35
Thanks to all contributors!

GitHub

Release v0.1.35 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.35">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.35</a> What's Changed Updated docs for release v0.1.34 by @AverageM...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.35

:emoji_k8s_loft: Drew Hudson-Viles

Abhay Krishna Arunachalam

2024-09-20 04:28:17

Hello folks, I have opened a PR to fix a Goss bug that caused our RHEL 8 and 9 qemu/raw builds to fail due to validation errors. I have updated the PR description with the necessary information about the bug. I would greatly appreciate some reviews on it, thank you!

:thank_you_icon: Marcus Noble

👍 mboersma

2024-09-20 17:53:39

@KJ has joined the channel

Omar

2024-09-22 06:49:20

@Omar has joined the channel

Slackbot

2024-09-23 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-23 15:39:49

Agenda empty - shall we still meet or skip? Anyone got anything to discuss?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-23 15:51:22

I think I'm going to skip. I could do with a rest as not been able to focus at all today 😩 If anyone decides to sync then ping me and I'll try to join.

❤️ mboersma

mboersma

2024-09-23 15:58:05

Sounds good to me, sorry to chime in late. I don't have anything for the agenda but if anyone does, I'll check in a bit and start the meeting if so.

👍 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-26 10:52:41

Is anyone building CAPA images that can pull from a private ECR registry?
If so - how are you including the ecr-credential-provider in your AMIs now that it's not included in-tree? (Related issue - )

Danny Seymour

2024-09-27 00:32:11

I'm actually getting all of this set up right now. I basically wrote a custom python script to generate packer vars including the correct ECR provider version and then wrote a role to install the provider.

Danny Seymour

2024-09-27 00:34:38

I elected to put the logic inside a Python script because I'm not good enough with Ansible to navigate the Github releases API with it and choose the appropriate version.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 06:15:38

I’m attempting to get it added into image-builder behind a Boolean toggle. Getting a bit tangled up with the ansible vars but hoping to have something today.

Danny Seymour

2024-09-27 07:22:50

Ooooo...I'd love to see that.

Danny Seymour

2024-09-27 07:23:11

If you want a tester, sign me up.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 07:23:32

Well... hopefully I get it working today 🤞 We're currently blocked on an upgrade without it so I need to figure something out.

Danny Seymour

2024-09-27 07:25:50

Heh...I know the feeling. We've been blocked on getting upgraded past 1.28.3 due to the whole VMWare acquisition led to the discontinuance of CAPI images. I'm trying to get a whole build pipeline automated so we can quit worrying about winding up outside the supported version window.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 07:26:49

Yeah, we build all our images too. But mainly because we wanted consistency across different providers that we support. It's constantly a work in progress it seems 🙈

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 10:39:13

Ok, at first glance, my changes are looking good:

/ # ls -la /host/opt/bin/ecr-credential-provider
-rwxr-xr-x    1 root     root        477197 Sep 27 09:08 /host/opt/bin/ecr-credential-provider
/ # ls -ls /host/var/usr/ecr-credential-provider/ecr-credential-provider-config
     4 -rw-r--r--    1 root     root           337 Sep 27 09:08 /host/var/usr/ecr-credential-provider/ecr-credential-provider-config
/ # cat /host/var/usr/ecr-credential-provider/ecr-credential-provider-config
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages: ['.dkr.ecr..amazonaws.com', '.dkr.ecr..amazonaws.com.cn']
    defaultCacheDuration: "12h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    env:
      - name: AWS_PROFILE
        value: "default"
/ # ps ax | grep /opt/bin/kubelet
 3225 root      0:02 /opt/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --healthz-bind-address=0.0.0.0 --v=2 --image-credential-provider-config=/var/usr/ecr-credential-provider/ecr-credential-provider-config --image-credential-provider-bin-dir=/opt/bin

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 10:39:33

Just need to get another team to test it out and make sure it actually works with ECR then I'll get a PR up.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 16:49:01

Confirmed working 😄 PR is up

GitHub

Add optional support for including ecr-credential-provider by AverageMarcus · Pull Request #1570 · kubernetes-sigs/image-builder

Change description Adds a new ansible role that can be optional enabled via the ecr_credential_provider boolean var that will install and configure the use of ecr-credential-provider to allow pull...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1570

Danny Seymour

2024-09-27 20:25:11

I did ask a question on the PR after it got merged. It's a bit of a nit, but I figure it's probably better to ask.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 20:30:29

I wasn’t actually aware it was available at that URL. I struggled to find docs about it. If that URL works the same then yeah it makes sense to use that instead. Do you want to open a PR for it?

Danny Seymour

2024-09-27 20:35:45

Sure

Danny Seymour

2024-09-27 20:36:13

It took me a while to find it too when I went searching.

Danny Seymour

2024-09-27 21:05:32

This is the first reference I could find for it related to sigs:

GitHub

k8s.io/artifacts/README.md at c651233068b2fed00fe801280ee067205efc9423 · kubernetes/k8s.io

Code and configuration to manage Kubernetes project infrastructure, including various *.<a href="http://k8s.io">k8s.io</a> sites - kubernetes/k8s.io

Original URL: https://github.com/kubernetes/k8s.io/blob/c651233068b2fed00fe801280ee067205efc9423/artifacts/README.md#artifact-promotion

Danny Seymour

2024-09-27 21:08:06

GitHub

cleanup: Use artifacts.k8s.io instead of storage.googleapis.com for ECR credential provider by thefirstofthe300 · Pull Request #1572 · kubernetes-sigs/image-builder

Change description The k8s-artifacts-prod is currently fronted by a proxy at <a href="http://artifacts.k8s.io">artifacts.k8s.io</a>. Assuming that there&#39;s the potential for S3 or Azure Blob Storage to ever be used as a backend in a...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1572

💙 Marcus Noble

:thank_you_icon: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-30 07:35:19

I was considering pushing out a new release with this change but I think I'm going to hold off until there's more changes that need releasing. Just, FYI 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-09-27 19:44:10

Image-builder v0.1.36 is now available:

This include the above change I was discussing to optionally include ecr-credential-providers.

GitHub

Release v0.1.36 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.36">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.36</a> What's Changed Update docs for release v0.1.35 by @mboersma ...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.36

:thank_you: mboersma

🎉 mboersma

Arnaud Pons

2024-10-01 07:30:21

@Arnaud Pons has joined the channel

Ian King’ori

2024-10-01 12:09:19

@Ian King’ori has joined the channel

TinaMor

2024-10-01 16:52:57

@TinaMor has joined the channel

Yongxiang Gao

2024-10-04 07:51:36

I try to build vSphere image with such command:

IB_OVFTOOL_ARGS="--allowExtraConfig" make build-node-ova-vsphere-ubuntu-2204-efi

During the build process, we can see following message:
==> vsphere-iso.vsphere: Power on VM...
==> vsphere-iso.vsphere: Waiting 10s for boot...
==> vsphere-iso.vsphere: HTTP server is working at
==> vsphere-iso.vsphere: Typing boot command...
==> vsphere-iso.vsphere: Waiting for IP...
==> vsphere-iso.vsphere: IP address: 10.20.34.127
==> vsphere-iso.vsphere: Using SSH communicator to connect: 10.20.34.127
==> vsphere-iso.vsphere: Waiting for SSH to become available...

The VM was 10.20.34.127 initially, however, very soon, it reboot and the IP change to something like 10.20.34.128, then "Waiting for SSH to become available..." will stuck there forever (because it's waiting for the old IP).

2004 doesn't have this issue.

How to avoid this 2204 specific issue?

Yongxiang Gao

2024-10-04 08:12:42

Same issue exists for the command:

image-builder build --os ubuntu --os-version 22.04 --hypervisor vsphere --release-channel 1-29 --vsphere-config vsphere-34.json --firmware efi

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 08:18:38

I don't have an answer myself but does this help at all?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 08:18:57

Looks like you might be able to force it to use a specific IP address at boot. Not sure how we solve this permanently though.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 08:19:43

Or possibly this:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 08:20:02

It might be we're not waiting long enough for it to detect the new IP address from vsphere

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 08:21:57

Try setting boot_wait to 300 in your packer vars and see if that helps at all. If it does we could try tweaking it until we find a low enough value that works.

Yongxiang Gao

2024-10-04 08:56:25

Thanks, I try following 3 ways in file packer/ova/packer-node.json:

change boot_wait to 300 at line 42

add at the front of line 38

add at the end of line 40

None of them helps.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 08:57:25

Can you confirm that in the vcenter console it is picking up the new IP address?

Yongxiang Gao

2024-10-04 08:59:44

Yes, in vcenter, the new IP shows up very soon.

Yongxiang Gao

2024-10-04 09:00:04

no 5 minute wait at all.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-04 09:00:51

Hmmmm... I thought that was how Packer discovered the IP. Maybe someone who knows more about vsphere can suggest something.

Danny Seymour

2024-10-04 23:03:39

It looks like this may be caused by the behavior of the DHCP server in use.

GitHub

Having the same issue as closed #127 · Issue #307 · hashicorp/packer-plugin-vsphere

I have tried every published workaround but have the same issue as #127 Nothing I have tried will keep the VM from pulling new IP on reboot. I am on the latest releases of everything. Is the bug th...

Original URL: https://github.com/hashicorp/packer-plugin-vsphere/issues/307

Danny Seymour

2024-10-04 23:12:36

Ah, that's one of the things that Marcus is getting at with the boot_wait.

Yongxiang Gao

2024-10-10 18:50:40

The trick doesn't help (the IP still changed) with the image-builder build from source if put in late-commands.
Instead, if put in early-commands, it helps:

images/capi# git diff packer/ova/linux/ubuntu/http/22.04.efi/user-data
diff --git a/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data b/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data
index 095d9cef3..ee9bb9705 100644
--- a/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data
+++ b/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data
@@ -20,6 +20,9 @@ autoinstall:
   # Disable ssh server during installation, otherwise packer tries to connect and exceed max attempts
   early-commands:
     - systemctl stop ssh
+    # Prevent DHCP release message from being sent on reboot
+    - iptables -A OUTPUT -p udp --dport 67 -j DROP

images/capi# IB_OVFTOOL_ARGS="--allowExtraConfig" PACKER_FLAGS="--var 'kubernetes_rpm_version=1.28.9' --var 'kubernetes_semver=v1.28.9' --var 'kubernetes_series=v1.28' --var 'kubernetes_deb_version=1.28.9-2.1'" make build-node-ova-vsphere-ubuntu-2204-efi
...
==> vsphere-iso.vsphere: IP address: 10.20.34.145
==> vsphere-iso.vsphere: Using SSH communicator to connect: 10.20.34.145
==> vsphere-iso.vsphere: Waiting for SSH to become available...      ** the IP stays at 10.20.34.145
...

However, is there side-effects?

Yongxiang Gao

2024-10-11 16:56:32

If there is side effect, what's the right way to fix the side effect?

Sriraman Srinivasan

2024-10-17 06:31:04

Try to increment ipsettletimeout and ipwaittimeout .

Yongxiang Gao

2024-10-21 06:27:58

@Sriraman Srinivasan Where to increase? Please share a diff.

Sriraman Srinivasan

2024-10-21 10:33:06

You can create a custom json file

Add ipsettletimeout to the json and set it to say 15m or adjust according to time for boot.

Add path of the json file to PACKERVARFILES environment and run OVA build

Sriraman Srinivasan

2024-10-21 13:04:46

ipsettletimout will make packer wait for specified time duration before it asks for the VM IP address.

:ty: Yongxiang Gao

Will Foster

2024-10-04 19:28:04

@Will Foster has joined the channel

leland knight

2024-10-06 00:18:08

@leland knight has joined the channel

leland knight

2024-10-06 02:01:58

I've got a question around image-builder-proxmox, where can I ask about the proxmox image builder?

==> proxmox-iso.ubuntu-2204:  => downloaded_iso_path/c968bbbeb22702b3f10a07276c8ca06720e80c4c.iso
==> proxmox-iso.ubuntu-2204: 501 for data too large
Build 'proxmox-iso.ubuntu-2204' errored after 2 minutes 31 seconds: 501 for data too large
==> Wait completed after 2 minutes 31 seconds
==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-2204: 501 for data too large

I saw an older posting with this error and it sounded like it had something to do with the Mime-type used when uploading.

Additional detailed added to a ticket:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-07 07:07:06

I've only used the Proxmox provider a little bit for some testing so might not be able to help but I do remember it being quite a pain to get set up.

Can you share what make target you're using and what vars you're providing (with anything sensitive redacted)

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-07 07:08:21

Also, do you have any load balancer or proxy or anything in front of your proxmox api that might be imposing the limit? (e.g. nginx)

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-07 07:10:22

I can't remember exactly but I think I ended up manually downloading the base iso into Proxmox prior to running image-builder

leland knight

2024-10-07 17:11:58

I also ended up manually uploading an iso to proxmox, and I thought it helped at first, but then didn't seem to. Figured I was just starting to hack at it rather than fix the actual issue.

I wonder if you might click on the link I provided, as I put a bunch of additional detail in the ticket 288.

leland knight

2024-10-10 02:40:57

answer:

501 is because I needed to add /api2/json

"using closed network connection" was due to 403

in my case this was difficult to diagnose because i was granting the correct access to an api token on the storage needed; but i hadn't granted the user the needed access which the api token came from; i added permissions to the user then the storage access worked

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-10 03:32:26

Are you able to suggest any improvements to our documentation?

leland knight

2024-10-10 03:35:14

I'm thinking about it ... figured I should probably first get things working then maybe go back and add something. So far I'm thinking:

it'd be nice if those two errors could be more descriptive so the person running into them could be faster to resolve them, i put a couple suggestions in the issue i closed

i went back to the proxmox instructions and it was clear to add permissions to the user (not the api token), simply i install proxmox a few days ago for the first time ... so that wasn't enough for me to avoid the issue i ran into; it would be nice to maybe have an additional step that lists out the permissions of the users and greps for the required permission ... something like that because as the person doing it i would have then realized i didn't make the user correctly

leland knight

2024-10-10 03:36:04

that's what i did when i fixed them; i listed out the user permissions; added the role; then listed them out again to see that they had changed

leland knight

2024-10-10 03:36:28

(cause at that point i wasn't sure if i had added the permissions before or not, just wasn't sure)

leland knight

2024-10-10 03:36:50

if i have time i'll put together a youtube video maybe, will let you know if i do

leland knight

2024-10-10 03:43:10

Perhaps there is an opportunity to make the process easier for folks in the future in the documentation, and perhaps in the error messages. Instead of 501 maybe "501, did you remember /api2/json?" Instead of "use of closed network connection" maybe "use of closed network connection (403)" or "use of closed network connection, see /var/log/pveproxy/access.log for more detail".

leland knight

2024-10-10 03:51:39

if someone forgets /api2/json, maybe it could just add it automatically ... or the URL environment variable could be checked early on to see if it has '/api2/json' and fail quick if it doesn't

leland knight

2024-10-10 04:43:11

also, how to specify which kubernetes version to build, the doc page () lists 4 env variables ... but i wonder if we actually just need one ... like KUBERNETES_SEMVER ... or do we need to specify all 4; when i run the build without specifying any KUBERNETES variables i don't really know what's going to happen? it will probably build the latest kubernetes, i assume? but don't know

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-10 06:55:27

That error message comes directly from Packer so not something we can really control in image-builder unfortunately. We could maybe do a check for the api2 path though. The example on the docs page does include the path though but maybe we need to make it clearer somehow.

If you have any suggestions on how to make those docs better we'd very much appreciate a PR! 😄 None of the maintainers of image-builder use Proxmox so we rely heavily on user contributions here.

I answered the kube version question in your other thread 🙂

Justin Holmes

2024-10-06 21:25:06

@Justin Holmes has joined the channel

Slackbot

2024-10-07 15:30:12

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-07 15:37:07

Agenda currently empty. If no one has anything they'd like to discuss then lets skip 🙂

mboersma

2024-10-07 15:49:39

I’ll have to miss it regardless today, sorry! Taking our animals to the vet for a checkup.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-07 15:50:41

ok lets skip 🙂

👍 mboersma

Drew Hudson-Viles

2024-10-07 16:40:51

Sorry, I've only just seen the time. The whole family is ill and the lack of sleep has made today just wizz by 😄

❤️ mboersma

mboersma

2024-10-07 16:48:01

Hang in there Drew!

🙏 Drew Hudson-Viles

richcase

2024-10-09 08:55:13

Should i be worried about these warnings in the image-builder logs? Skipped '/run/netplan' path due to this access issue

Drew Hudson-Viles

2024-10-10 19:31:02

I've not come across this one myself but it may just be the path isn't available if it's skipping over. As long as the image is coming online on boot, I wouldn't be too concerned.

richcase

2024-10-11 15:56:34

Thanks @Drew Hudson-Viles. The image does come online.

👍 Drew Hudson-Viles

leland knight

2024-10-10 02:42:38

Where does "proxmox-iso.ubuntu-2404" come from? I mean, where is it built so I can see what's in there maybe tweak it a little. Looks like there is a bug and I was interested to look into it a bit. (Or is that the new vm being built? It said ISO so thought it might have been an iso)

==> proxmox-iso.ubuntu-2404: Error creating VM: format can only be one of the following values: cow,cloop,qcow,qcow2,qed,vmdk,raw
Build 'proxmox-iso.ubuntu-2404' errored after 3 minutes 31 seconds: Error creating VM: format can only be one of the following values: cow,cloop,qcow,qcow2,qed,vmdk,raw

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-10 06:46:43

It should pull it from here I think:

GitHub

image-builder/images/capi/packer/proxmox/ubuntu-2404.json at 2f188e738f961730645269fe942cfcbb0925db7a · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/2f188e738f961730645269fe942cfcbb0925db7a/images/capi/packer/proxmox/ubuntu-2404.json#L8

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-10 06:47:06

If I understand your question correctly.

leland knight

2024-10-10 22:22:26

I'm was hoping to troubleshoot the error maybe ... but I'm too in the dark on this one. Thank you. I'll just hang out and hope the issue 1579 gets some attention.

leland knight

2024-10-10 04:33:55

How can I automatically set the PACKER kubernetes versions? I feel like I'm reinventing the wheel here ... and not sure how to get the DEB version:

# parse and set kubernetes env vars
echo "PACKER_FLAGS: $PACKER_FLAGS"
export VERSION=v1.31.1

#KUBERNETES_RPM_VERSION=1.29.6
#KUBERNETES_SEMVER=v1.29.6
#KUBERNETES_SERIES=v1.29
#KUBERNETES_DEB_VERSION=1.29.6-1.1

KUBERNETES_RPM_VERSION=$(echo $VERSION | cut -d 'v' -f 2)
KUBERNETES_SEMVER=$VERSION
KUBERNETES_SERIES=$(echo $VERSION | cut -d '.' -f 1).$(echo $VERSION | cut -d '.' -f 2)
KUBERNETES_DEB_VERSION=$(echo $VERSION | cut -d 'v' -f 2)-1.1

export PACKER_FLAGS="--var kubernetes_rpm_version=$KUBERNETES_RPM_VERSION --var kubernetes_semver=$KUBERNETES_SEMVER --var kubernetes_series=$KUBERNETES_SERIES --var kubernetes_deb_version=$KUBERNETES_DEB_VERSION"

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-10 06:50:16

You can define your own vars JSON file and pass that in when calling image-builder.
E.g. I have a vars.json that looks something like this (with the env vars populated via shell fist):

{
  "ssh_clear_authorized_keys": "true",
  "kubernetes_deb_version": "${KUBERNETES_VERSION}-00",
  "kubernetes_rpm_version": "${KUBERNETES_VERSION}-0",
  "kubernetes_semver": "v${KUBERNETES_VERSION}",
  "kubernetes_series": "v${VERSION_MAJOR}.${VERSION_MINOR}",
  "enable_containerd_audit": "true",
  "ecr_credential_provider": "true"
}

And then set the environment variable PACKERVARFILES to the location of that vars.json when calling image-builders Make targets

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-10 06:51:31

Unfortunately its not quite as easy as just saying "give me Kubernetes v1.31" as the different platforms/OSs have different ways of pulling them in and we don't have the capacity in the project to keep an up-to-date mapping of those versions.

Peter Lindblom

2024-10-11 08:33:31

@Peter Lindblom has joined the channel

mcbenjemaa

2024-10-11 12:03:10

Hey there,

I have new PR:

/cc @leland knight
can you test my PR on your setup, to double check.
It is actually working for me.

leland knight

2024-10-11 15:22:13

It's getting further along now. At the moment I'm seeing this message:

Error getting SSH address: 500 QEMU guest agent is not running

leland knight

2024-10-11 15:52:55

With proxmox build now seeing:

Error getting SSH address: 500 QEMU guest agent is not running

(dhcp is enabled and works)

The console shows:

mcbenjemaa

2024-10-11 16:25:19

The packer VM couldn't access the packer HTTP server.

You need to run this in Proxmox VM (same network) for example, so you're sure that communication works

leland knight

2024-10-11 16:29:18

are we sure the qemu guest agent is getting installed by default?

leland knight

2024-10-11 16:32:17

i'll verify i'm able to reach the port from a remote system...

leland knight

2024-10-11 16:38:11

all tests showing network connectivity is good and no firewall issues; able to curl the port image-builder opens up ; gives me a 403, but that's ok, communication is good

leland knight

2024-10-11 16:39:00

i stopped a firewall to be completely sure, so i'll let it run for a bit with the 500 ... but i suspect the communication should happen pretty quick

leland knight

2024-10-11 17:00:41

still says waiting on 500, but the console isn't showing the startup screen anymore, i see automation in there so it seems to be working ... acting like its stuck on 'installing kernel' ... fingers crossed it works

leland knight

2024-10-11 17:14:16

things were looking good, finished this time but with an error:

2024/10/11 10:09:01 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/11 10:09:01 [INFO] 0 bytes written for 'stdin'
    proxmox-iso.ubuntu-2204:
    proxmox-iso.ubuntu-2204: TASK [Gathering Facts] *
    proxmox-iso.ubuntu-2204: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /home/travis/.ansible/tmp/ansible-local-350766c889qert/tmph0vrydqs /tmp/.ansible/ansible-tmp-1728662940.5550468-350781-30051397384999/AnsiballZ_setup.py:\n\n"}
    proxmox-iso.ubuntu-2204:
    proxmox-iso.ubuntu-2204: PLAY RECAP 
    proxmox-iso.ubuntu-2204: default                    : ok=1    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
    proxmox-iso.ubuntu-2204:
2024/10/11 10:09:01 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/11 10:09:01 shutting down the SSH proxy
2024/10/11 10:09:01 [INFO] (telemetry) ending ansible
==> proxmox-iso.ubuntu-2204: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox-iso.ubuntu-2204: Stopping VM
==> proxmox-iso.ubuntu-2204: Deleting VM
2024/10/11 10:09:05 [INFO] (telemetry) ending ubuntu-2204
==> Wait completed after 23 minutes 34 seconds
2024/10/11 10:09:05 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/10/11 10:09:05 machine readable: proxmox-iso.ubuntu-2204,error []string{"Error executing Ansible: Non-zero exit status: exit status 2"}
==> Builds finished but no artifacts were created.
2024/10/11 10:09:05 [INFO] (telemetry) Finalizing.
Build 'proxmox-iso.ubuntu-2204' errored after 23 minutes 34 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 23 minutes 34 seconds

==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-2204: Error executing Ansible: Non-zero exit status: exit status 2

==> Builds finished but no artifacts were created.
2024/10/11 10:09:05 waiting for all plugin processes to complete...
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/hashicorp/proxmox/packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64: plugin process exited
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/11 10:09:05 /usr/bin/packer: plugin process exited
2024/10/11 10:09:05 /usr/bin/packer: plugin process exited
2024/10/11 10:09:05 /usr/bin/packer: plugin process exited
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/YaleUniversity/goss/packer-plugin-goss_v3.2.12_x5.0_linux_amd64: plugin process exited
make: * [Makefile:593: build-proxmox-ubuntu-2204] Error 1

mcbenjemaa

2024-10-11 17:23:18

Its just some ansible issue.
Make sure everything fine.

leland knight

2024-10-11 17:36:15

Is the end result a randomly named iso in the storage?

mcbenjemaa

2024-10-11 17:36:42

Proxmox template

leland knight

2024-10-11 17:37:11

it should be named proxmox template?, or ... you are saying the randomly named iso is the proxmox template?

first time i'm using this, not even sure what the result is supposed to be, not sure if this is a successful run or not since i got the ansible error

leland knight

2024-10-11 17:38:50

I would have expected something like, "Success! Generated xxx.iso".

mcbenjemaa

2024-10-11 17:39:51

What are you talking about?
What ISO
It should be Virtual machine template in Proxmox.

leland knight

2024-10-11 17:45:38

I don't see a template, I think after the ansible error it just shutdown the vm and deleted it.

==> proxmox-iso.ubuntu-2204: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox-iso.ubuntu-2204: Stopping VM
==> proxmox-iso.ubuntu-2204: Deleting VM

mcbenjemaa

2024-10-11 17:46:36

Thats because the build was failed

leland knight

2024-10-11 18:02:13

I don't know how to troubleshoot this, so just put in another issue ...

mcbenjemaa

2024-10-11 18:10:57

Please create only one issue.
And comment there.

mcbenjemaa

2024-10-11 18:12:41

Again it seems some ssh or scp is failing in your setup.
Try to check your firewall whatsoever
or anything you need to add to ansible ssh or scp

leland knight

2024-10-12 01:14:48

The firewall is still disabled, so that shouldn't be the issue ... I'm not sure what else would be needed

leland knight

2024-10-12 01:16:25

What is it trying to transfer and from where to where?

Mike Tritabaugh

2025-06-11 14:58:57

Adding my two cents in case anyone else runs into this error and finds this thread…

I got this error after successfully building multiple images. Nothing obvious had changed, I even tried rebuilding images that previously worked.

I resolved it by updating the ubuntu-2404.json file. I changed the boot command to hard code my laptop’s IP address (where I’m running the make command).

  "boot_command_prefix": "clinux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=:{{ .HTTPPort }}/24.04/'initrd /casper/initrdboot",

My uneducated hypothesis:
After setting PACKER_LOG=1, I noticed that the http server was starting on 0.0.0.0 . This means all interfaces, but I suspect this IP is being passed to the build VM as a callback, which obviously wouldn’t work. After making the change above, the builds completed successfully. I’m assuming something changed in my environment (it had been a couple weeks since I ran a build) that caused the http server to change the bind IP used.

Or, I could be completely wrong. 🤷‍♂️

leland knight

2024-10-11 15:54:14

[ MESSAGE TEXT EMPTY ]

image.png

leland knight

2024-10-11 16:13:41

Issue filed:

richcase

2024-10-11 18:01:50

If anyone is free i could do with a review on this change:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-11 18:29:39

Thanks! I thought it was a duplicate of https://github.com/kubernetes-sigs/image-builder/pull/1586 at first but looks like it’s different Ubuntu versions. Do you know if any others need changing?

GitHub

fix(capi-ubuntu24): pin cloud-init deb package version by dntosas · Pull Request #1586 · kubernetes-sigs/image-builder

Change description Newer version published on Ubuntu upstream stable channel introduced differences in cloud-init behavior making AMI builds stale. Several issues are around community/projects and...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1586

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-11 18:30:06

Is this just an issue on AWS?

richcase

2024-10-11 18:35:42

Its just an AWS issue as far as i know as CAPA does some hacky things with cloud-init

richcase

2024-10-11 18:36:07

We're working on a longer term solution so that we don't need to pin to an old version.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-11 18:48:27

👍 great! Are you needing a new release of image-builder putting out then?

richcase

2024-10-11 18:49:12

I'm just using main at the moment.

richcase

2024-10-11 18:49:34

But thinking about it we should pin our build pipeline to a version.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-11 18:50:06

Ok. If you’re ok for now I can hold off until Monday to get a new release out.

richcase

2024-10-11 18:50:28

Sounds good

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-11 18:50:51

Unless I find a bit of time tomorrow. I might kick off the process as there’s been a few fixes the past week.

👍 richcase, leland knight

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-12 13:10:21

😞 Looks like I'm actually unable to push a new release out right now -

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)

Hey, not sure if this is the right place to ask this but I've got an issue with a pre-submit Prow job failing. Is there anyone with permission that could attempt re-running it for me? It doesn't seem like I have the ability to. <a href="https://prow.k8s.io/?repo=kubernetes-sigs%2Fimage-builder&type=postsubmit&job=post-image-builder-push-images">https://prow.k8s.io/?repo=kubernetes-sigs%2Fimage-builder&type=postsubmit&job=post-image-builder-push-images</a>

Original URL: https://kubernetes.slack.com/archives/CCK68P2Q2/p1728733103390359

:ack: richcase

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-12 18:49:30

Image-builder v0.1.37 is now available:

Some notable changes:

This release fixes for building Ubuntu images on CAPA by pinning the version of cloud-init deb

The default Kubernetes version is updated to v1.30.5

The ability to set the isofile or isourl when building Proxmox images proxmox

Adds Windows Server 2025 to Azure azure

Some refactoring and cleanup of the code 🧹

Thanks to all contributors! 🎉

GitHub

Release v0.1.37 · kubernetes-sigs/image-builder

Image Builder v0.1.37

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.37

🎉 Drew Hudson-Viles, leland knight, richcase

leland knight

2024-10-12 22:11:04

I'd like to run image builder via a gitlab runner. I know what IP it will use, but being that its running in kubernetes its exposed via a service, or even better an ingress and I have the FQDN. Where can I specify the IP to use or specify FQDN? or otherwise tell image-builder to use the loadbalancerip when it calls back to the image-builder http port?

Submitted as a feature request

Continuing to search through docs to see if the feature already exists and I just haven't found the right env vars yet.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 16:25:47

Image-builder v0.1.38 is now available:

⚠️ Important
This release contains fixes for two CVEs - CVE-2024-9486 and CVE-2024-9594 (see kubernetes/kubernetes#128006 & kubernetes/kubernetes#128007 for more details).
It is highly recommended to update your version of image-builder and re-build all your VM images.

GitHub

Release v0.1.38 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.38">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.38</a> Important This release contains fixes for two CVEs - CVE-2024...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.38

:thank_you: mboersma, Drew Hudson-Viles, Abhay Krishna Arunachalam, mcbenjemaa

🎉 mboersma, Drew Hudson-Viles

:partyk8s: mboersma, Drew Hudson-Viles, leland knight

:kubernetes_intensifies: Drew Hudson-Viles

Abhay Krishna Arunachalam

2024-10-14 17:38:11

Hello Image-builder maintainers, we just migrated to v0.1.38 to fix the CVEs and our Nutanix builds started failing with the error

==> nutanix: Waiting for SSH to become available...
==> nutanix: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> nutanix: Force deleting virtual machine...
    nutanix: Virtual machine successfully deleted

I am actively looking into it, but would appreciate if any of you folks might know what's causing the builds to act up. Maybe some hard-coded credentials needs to be removed?

Abhay Krishna Arunachalam

2024-10-14 17:39:51

Seeing the same for Vsphere ISO

==> vsphere-iso.vsphere: Waiting for SSH to become available...
==> vsphere-iso.vsphere: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

Drew Hudson-Viles

2024-10-14 17:49:48

I'm wondering if it's this change and that uuid is being overridden.

https://github.com/kubernetes-sigs/image-builder/pull/1596/files#diff-19301ae03119dcb0a5ed81f1e5839a3a26725e486cb89a3cc3e70c1b1df0b159R160

Can you confirm your vars for me? Do you have a password set in there that configures this value already?

GitHub

Set random ssh password for builder by AverageMarcus · Pull Request #1596 · kubernetes-sigs/image-builder

For providers that don&#39;t have a built-in support for random ssh keys this change makes sure that the default builder password used during the image build process is a randomly generated value. ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1596/files#diff-19301ae03119dcb0a5ed81f1e5839a3a26725e486cb89a3cc3e70c1b1df0b159R160

Abhay Krishna Arunachalam

2024-10-14 17:51:50

no we don't set the password manually, have always resorted to the builder/builder default, and now we've taken in this commit without any patches to it

Abhay Krishna Arunachalam

2024-10-14 17:59:27

Just noticed that QEMU and raw image builds also followed suit with the same failure reason

Drew Hudson-Viles

2024-10-14 18:00:18

Hmmm, ok. This was tested as working so we'll have to do some digging into it.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 18:00:40

I'm just kicking off a CAPV build in my environment to see if I'm seeing the same.

Drew Hudson-Viles

2024-10-14 18:00:42

Sorry I'm phone troubleshooting at the moment but should be back at my pc later to assist where I can.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 18:01:14

@Abhay Krishna Arunachalam Do you use the container image?

Abhay Krishna Arunachalam

2024-10-14 18:03:44

no we use a different container image which contains the same versions of components as SCL image-builder has in its ensure-**.sh scripts.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 18:04:43

Ah ok, but same version of Packer, etc. yes?

:yes: Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-10-14 18:04:46

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 18:07:59

==> vsphere-iso.vsphere: Creating VM...
==> vsphere-iso.vsphere: Customizing hardware...
==> vsphere-iso.vsphere: Mounting ISO images...
==> vsphere-iso.vsphere: Adding configuration parameters...
==> vsphere-iso.vsphere: Set boot order temporary...
==> vsphere-iso.vsphere: Power on VM...
==> vsphere-iso.vsphere: Waiting 3m0s for boot...
==> vsphere-iso.vsphere: Typing boot command...
==> vsphere-iso.vsphere: Waiting for IP...
==> vsphere-iso.vsphere: IP address: 10.10.222.89
==> vsphere-iso.vsphere: Using SSH communicator to connect: 10.10.222.89
==> vsphere-iso.vsphere: Waiting for SSH to become available...
==> vsphere-iso.vsphere: Connected to SSH!
==> vsphere-iso.vsphere: Provisioning with shell script: ./packer/files/flatcar/scripts/bootstrap-flatcar.sh

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 18:09:49

I'm not able to reproduce this at least with my setup.
Are you able to share what Make target and what the final combination of user vars and environment varibles are for you're vspehre build (with anything sensitive redacted)

Abhay Krishna Arunachalam

2024-10-14 18:50:46

Running this make target

make -C image-builder/images/capi build-node-ova-vsphere-rhel-8

Christophe Jauffret

2024-10-14 18:57:45

Hello , quickly check and this is expected it is no more working
need to think how we can fix it

Abhay Krishna Arunachalam

2024-10-14 19:03:48

yeah the build before this change passed for us

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 19:11:11

Can you check if setting ssh_password to something yourself allows it to build or not?

Abhay Krishna Arunachalam

2024-10-14 19:38:47

kicked off a test build setting the password to a hardcoded string other than builder

Abhay Krishna Arunachalam

2024-10-14 19:40:49

Quick question, won't we also need to change this and other locations since the password is not builder anymore?

GitHub

image-builder/images/capi/packer/ova/linux/ubuntu/http/22.04/user-data at e59e76be444eba93a4adaf000447fbefea2fd57a · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/e59e76be444eba93a4adaf000447fbefea2fd57a/images/capi/packer/ova/linux/ubuntu/http/22.04/user-data#L74

Christophe Jauffret

2024-10-14 19:50:49

yes

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 19:50:50

Oh crap you might be right! I had no idea we hardcoded the password in Ubuntu like that!

Christophe Jauffret

2024-10-14 19:51:03

cloudinit is used to inject password everywhere

Christophe Jauffret

2024-10-14 19:51:12

now we need to templatize the cloud-init content

Abhay Krishna Arunachalam

2024-10-14 19:52:50

also for RHEL in the kickstart file

Abhay Krishna Arunachalam

2024-10-14 19:53:39

but still, these files will be served and the user in the autoinstall will be created only after the SSH communicator goes through and then access the autoinstall files right? So it shouldn't affect the initial SSH connection?
Okay never mind, the boot command is typed before the SSH connection so I think it must definitely look at the builder user being created in the autoinstall file

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 20:06:12

@Abhay Krishna Arunachalam what happens if you set the password back to builder in your vars?

Abhay Krishna Arunachalam

2024-10-14 20:24:22

Running a build with builder and a separate one with hello. Will keep the thread posted

👍 Drew Hudson-Viles, Marcus Noble

Abhay Krishna Arunachalam

2024-10-14 20:26:14

I'm curious how it worked in your case though, becuase it seems the SSH password should be the same as what's being created during autoinstall, but builder and the randomly generated UUID would be a mismatch

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-14 20:27:25

I use flatcar which uses ignition rather than the user data

Abhay Krishna Arunachalam

2024-10-14 20:29:20

ah I see. If I kick off a presubmit on image-builder through some dummy change, that would kick off an Ubuntu OVA right?

Abhay Krishna Arunachalam

2024-10-14 20:31:28

In terms of a fix, I think the UUID thing is internal to Packer so it's probably never printed or returned

so I think what we need to do is to generate it outside of Packer (think Makefile or script) and replace in the JSON as well as the user-data

Abhay Krishna Arunachalam

2024-10-14 20:36:58

was the pull-ova-all removed from Prow? I think that might have caught this

Abhay Krishna Arunachalam

2024-10-14 20:44:25

tried this on a linux box

$ uuidgen
04c84fad-871c-4bdf-9307-4ab8f16e5993

Abhay Krishna Arunachalam

2024-10-14 20:49:16

Update: the build with password set to builder passed, but the hello one is stuck waiting for SSH

mboersma

2024-10-14 21:08:18

Yes, pull-ova-all was removed during the community infra migration. We're still waiting for a replacement, see

Abhay Krishna Arunachalam

2024-10-14 21:38:56

Ah I see. Does the fix I suggested above make sense/sound feasible?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 06:01:29

Yeah that sounds feasible to me

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 08:38:51

@Abhay Krishna Arunachalam I just saw your PR! Thank you so much! I’ll take a look at it hopefully later when I’m able to (currently travelling to a conference). Just wanted to check - have you tested if the change works when building a Flatcar image too?

🙌 Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-10-15 08:40:00

so flatcar already has a mkpasswd command wired into a sed command, which i chose not to touch

Drew Hudson-Viles

2024-10-15 08:40:46

You legend, I was about to start looking at the issue this morning! 😄 I'll do some testing on a few things my side and give it my stamp then if all is well from the thing I can test.

💙 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 08:40:58

🎉 Great! Hopefully I can get it reviewed this morning and get it merged in. I really appreciate the effort!

Abhay Krishna Arunachalam

2024-10-15 08:42:29

Thank you all, appreciate it! I'm also going to test it out on my end since we have a Nutanix/QEMU/Raw/Vsphere testing bed in our CI

Drew Hudson-Viles

2024-10-15 08:43:46

Perfect. I can do the QEMU, Raw and OpenStack to make sure all is working on that side with this change too. (Never hurts to have a couple of perspectives with regards to the QEMU & RAW).

Drew Hudson-Viles

2024-10-15 09:18:55

I'm having a couple issue building QEMU atm, not sure if the envsubst is working as expected in terms of the file it outputs (still contains the $ENCRYPTEDSSHPASSWORD var) - I am flipping between work and this though might be missing something there 😉

Also we should probably consider gitignoring the generated files - i'll stick a note in the issue for that though.

Abhay Krishna Arunachalam

2024-10-15 09:19:57

yeah I had planned to put them in the gitignore too

👍 Drew Hudson-Viles

Abhay Krishna Arunachalam

2024-10-15 09:20:56

Also I might just replace envsubst with sed, since it requires you to have the gettext package installed

Drew Hudson-Viles

2024-10-15 09:35:16

I was wondering if sed might be a better option

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 09:35:27

Either replace with sed or have the binary downloaded into the local bin directory as part of the deps.

Abhay Krishna Arunachalam

2024-10-15 09:37:48

sed works, tested locally

🙌 Drew Hudson-Viles

🎉 Marcus Noble

Christophe Jauffret

2024-10-15 09:41:45

ideal solution would be to migrate the entire image-builder project to packer hcl format so we can use all the built-in function and stop using this kind of hacks 😉
i don’t know if there is plan for that

in all cases the above approach would need some additional modification for platform who are using builder who need to inject userdata directly inside packer config

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 09:44:36

Migrating to HCL is a massive undertaking unfortunately and no one has offered / been able to invest the time into doing it. And as we are sorely lacking in testing its risky to make such a large change to the project as we can't be sure we don't break someones use case. 😞

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 09:45:03

We do want to do it though. It's just not easy 😞

Christophe Jauffret

2024-10-15 09:45:36

yes i totally understood, but for sure one day in the future we will have no more choice

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 09:45:56

Yeah 😞

Drew Hudson-Viles

2024-10-15 09:57:01

I did actually test doing just the OpenStack builder and it's MUCH nice in the HCL format but yeah, huge task to do it for just that one - the whole project would be a lot of work - I'd like to do it but so many edge cases and "hacks" would have to be considered

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 10:07:37

PR looks good to me. I'll leave with @Drew Hudson-Viles to review as he's in a position to test it 🙂 I'll add my lgtm

Abhay Krishna Arunachalam

2024-10-15 10:10:09

Thanks a lot for the quick review!! blod-tada I've put the PR on hold until I have had a chance to test it comprehensively on my end. But I should be able to get it merged today and y'all can then cut a release.

👍 Marcus Noble

Abhay Krishna Arunachalam

2024-10-15 12:15:51

Also realized I need to fix it for Nutanix as well, since currently we have the userdata as a hardcoded base64 string in the packer config file (example), which resolves to

#cloud-config
users:
  - name: builder
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    shell: /bin/bash
chpasswd:
  list: |
    builder:builder
  expire: False
ssh_pwauth: True

Abhay Krishna Arunachalam

2024-10-15 15:43:54

Fixed Nutanix by adding a static cloud-init template which is base64-encoded and set as the user_data string during build time

Tested it on our CI and it works as expected

Abhay Krishna Arunachalam

2024-10-15 17:20:29

Not sure why the Azure presubmit is failing on the PR, is it unrelated?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 17:21:59

I think that is currently broken and @mboersma is working on removing them from CI.

:ack: Abhay Krishna Arunachalam

Drew Hudson-Viles

2024-10-15 17:23:25

Ftr, I'm still testing things my side. I've been so busy today I've not had much time to go through it all properly I'm afraid. Last time I checked qemu didn't work but haven't synced the branch since this morning.

Abhay Krishna Arunachalam

2024-10-15 17:25:08

i tested nutanix/qemu/raw/vsphere builds on our CI and they all passed

🎉 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 17:26:33

I haven't had a chance to look since this morning and wont now until Thursday. I trust Drew can handle it though 😄 He knows more about these providers than I do

Abhay Krishna Arunachalam

2024-10-15 18:47:47

@mboersma thanks for merging this! Helped get the Azure presubmits passing on my PR

GitHub

Remove windows-2019 and -2025 targets from CI by mboersma · Pull Request #1604 · kubernetes-sigs/image-builder

Change description Removes the Azure windows-2019 and windows-2025 targets from CI runs, since they are both failing and blocking unrelated PRs from merging. Related issues Refs Azure windows-2019...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1604

👍 mboersma

mboersma

2024-10-15 19:30:18

Good deal! Hopefully we can keep CI greener, it's been kind of flaky.

Abhay Krishna Arunachalam

2024-10-15 20:00:52

Nutanix CI builds

Screenshot 2024-10-15 at 11.54.12 PM.png

Screenshot 2024-10-15 at 11.54.37 PM.png

Abhay Krishna Arunachalam

2024-10-15 20:01:19

QEMU CI builds

Screenshot 2024-10-16 at 12.24.56 AM.png

Abhay Krishna Arunachalam

2024-10-15 20:03:48

Raw CI builds

Screenshot 2024-10-16 at 12.26.56 AM.png

Screenshot 2024-10-16 at 12.28.42 AM.png

Screenshot 2024-10-16 at 12.27.35 AM.png

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 20:11:19

Look at all that green! 😍

💚 Abhay Krishna Arunachalam, mboersma

Abhay Krishna Arunachalam

2024-10-15 20:12:43

Vsphere CI builds

Screenshot 2024-10-16 at 12.26.12 AM.png

Screenshot 2024-10-16 at 12.42.15 AM.png

Abhay Krishna Arunachalam

2024-10-15 20:17:49

Sorry for the spam! I added my image-builder PR as a patch in eks-anywhere-build-tooling and thought I'd show the CI results from the builds kicked off after the patch merged

GitHub

Add patch to set SSH password dynamically during image builds by abhay-krishna · Pull Request #3912 · aws/eks-anywhere-build-tooling

Pulling in kubernetes-sigs/image-builder#1607 as a patch until it&#39;s merged and released upstream. We need this to fix an SSH password bug introduced in image-builder v0.1.38. /hold By submittin...

Original URL: https://github.com/aws/eks-anywhere-build-tooling/pull/3912

Abhay Krishna Arunachalam

2024-10-16 16:10:33

@Drew Hudson-Viles thanks for the review! I have addressed your gitignore comment, are you good with that approach?

Drew Hudson-Viles

2024-10-16 16:18:49

Yeah that's absolutely fine. I think I just missed the push where you added it 🙂

Drew Hudson-Viles

2024-10-16 16:19:19

I've focussed on testing each time one comes in rather than reading them tbh. I've done a full read through and review now though so yeah, cracking work buddy!

🙌 Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2024-10-16 16:19:25

Great! I see you've held it for other reviewers, thanks for the approval!

Drew Hudson-Viles

2024-10-16 16:19:37

Yeah I didn't want to be the only one firing it through 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-16 16:23:59

I should be able to take a look in the morning if no one else gets to it before then. We can then get the released pushed out. 🙂

✅ Drew Hudson-Viles

:ty3: Abhay Krishna Arunachalam

Drew Hudson-Viles

2024-10-16 17:45:46

Thanks for approving that @mboersma! I'll look into getting a release done tomorrow if I can get time in the morning.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-16 17:59:44

I can do a release first thing if you want. Just need the PRs approving. 🙂

👍 Drew Hudson-Viles, mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 07:55:54

Right, starting the release process now 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:03:14

😅 seems to be down and causing the build to fail

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:03:38

or maybe just really slow 😕

Abhay Krishna Arunachalam

2024-10-17 08:10:52

is the tag creation alone the trigger for the postsubmit?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:12:42

Yeah

Abhay Krishna Arunachalam

2024-10-17 08:12:54

In the past I have been able to re-trigger postsubmits by re-delivering the webhook payload corresponding to a PR merge from the repo settings. I'll admit it's not the ideal solution and I'm not sure if there are any side effects because tags are involved

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:13:03

oh good idea!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:13:29

Oh, Prow isn't configured to use webhooks in the repo 🤦‍♂️

Abhay Krishna Arunachalam

2024-10-17 08:13:36

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:13:47

Yeah, I need someone with more power than me it seems 😅

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 08:15:19

I don't understand why these have been so flakey recently. This is the third release in the past week that has failed initially but I haven't had any problems the previous ~1 year 🤨

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:22:19

Looks like someone trigger the re-run for me and its now passed so continuing with the release

🙌 Abhay Krishna Arunachalam

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:31:52

@Drew Hudson-Viles

GitHub

Image promotion for scl-image-builder v0.1.39 by AverageMarcus · Pull Request #7426 · kubernetes/k8s.io

Image promotion for scl-image-builder v0.1.39 This is an automated PR generated from kpromo kpromo pr --fork AverageMarcus --project scl-image-builder --reviewers &quot;@AverageMarcus @jsturtevant ...

Original URL: https://github.com/kubernetes/k8s.io/pull/7426

✅ Drew Hudson-Viles

Drew Hudson-Viles

2024-10-17 11:35:26

Thanks for taking that on. I'm a 1 man band this week so time is short.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:37:14

No worries 🙂 You feeling any better today?

Drew Hudson-Viles

2024-10-17 11:37:45

Yeah much better thanks. Still not 100% but functional non-the-less!

❤️ mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:37:58

Oh good! 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:43:51

And docs PR:

GitHub

Update docs for v0.1.39 by AverageMarcus · Pull Request #1613 · kubernetes-sigs/image-builder

Change description Related issues Fixes # Additional context

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1613

✅ Drew Hudson-Viles

mboersma

2024-10-17 15:52:18

Thanks for doing the release!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 08:30:46

☝️ Following on from this issues (Sorry 😞) I've created an issue to track our progress towards testing as much of the providers and OSs we support in image-builder as possible. Please take a look and let me know if you have any comments or suggestions.

GitHub

Testing of all providers and distros · Issue #1605 · kubernetes-sigs/image-builder

Is your feature request related to a problem? Please describe. We've had a few instances now where changes introduced issues into existing targets, e.g. #881 #879 #828 and most recently #1596 cause...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1605

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 08:33:08

/cc @mboersma as I know you've been doing a fair bit of work related to this recently 🙂 Hopefully I haven't missed anything.

:thank_you: mboersma

Christophe Jauffret

2024-10-15 09:49:00

Concerning Nutanix we have an internal daily test on master branch

and the result this morning was not nice thanks to #1596 😰

image.png

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-15 10:05:36

That's internal right? Not something we could expose to PRs on image-builder?

Christophe Jauffret

2024-10-15 10:06:14

not for now

:ack: Marcus Noble

leland knight

2024-10-15 23:10:18

I think maybe the packer-plugin-sdk is where the http callback server is being created, I moved the ticket over to that github project:

martin.choe

2024-10-17 04:08:59

@martin.choe has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:43:01

Image-builder v0.1.39 is now available: 🎉

Thanks to all contributors! 💙

GitHub

Release v0.1.39 · kubernetes-sigs/image-builder

Image Builder v0.1.39

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.39

:kubernetes_intensifies: Drew Hudson-Viles, Abhay Krishna Arunachalam, Danny Seymour

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 11:49:31

I'd also like to say a big thank you to @Abhay Krishna Arunachalam for doing so much work to fix the ssh password issue for non-ignition distros! 💙 Huge help and something I was personally stressing about on Monday 😅

🎉 Jacob Weinstock, bavarianbidi

Abhay Krishna Arunachalam

2024-10-17 12:09:25

Happy to contribute, thank you for all the support in reviewing, testing and getting it merged! k8s-heart

karine.santos

2024-10-17 14:58:35

@karine.santos has joined the channel

karine.santos

2024-10-17 15:02:39

Hi everyone! I have a clone of the image builder and need to know the release of this clone. Where can I find this information in the repository?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-17 15:03:43

What do you mean by a clone of image builder? Of the git repo? You should be able to check the git commit to see what your copy is at.

leland knight

2024-10-17 15:46:52

@Marcus Noble or @mcbenjemaa when you get a minute, could you test out my proxmox-related pull request? Looks like the couple folks working on it don't have an environment setup to test with:

Abhay Krishna Arunachalam

2024-10-17 18:57:41

Small fix for the Nutanix gitignore

Sorry missed this in the other PR!

GitHub

Fix user-data filename in Nutanix gitignore by abhay-krishna · Pull Request #1614 · kubernetes-sigs/image-builder

Fix user-data filename in Nutanix gitignore to use the generated filename instead of the checked-in file.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1614

:+2: Drew Hudson-Viles

fad3t

2024-10-18 11:27:10

I'm having errors when building on Nutanix. Probably related to the random password thing.
I see the username and password are hardcoded in the user data here, is this expected?

GitHub

image-builder/images/capi/packer/nutanix/ubuntu-2204.json at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/nutanix/ubuntu-2204.json#L7

fad3t

2024-10-18 11:27:39

and the error I get:

nutanix: output will be in this color.
==> nutanix: Creating Packer Builder virtual machine...
    nutanix: Virtual machine ubuntu-2204-kube-v1.30.5 created
    nutanix: Found IP for virtual machine: 10.10.141.63
==> nutanix: Using SSH communicator to connect: 10.10.141.63
==> nutanix: Waiting for SSH to become available...
==> nutanix: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> nutanix: Force deleting virtual machine...

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-18 11:27:49

Have you updated to v0.1.39?

fad3t

2024-10-18 11:27:54

yes

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-18 11:28:22

😞 Damn! i haven't got access to a nutanix environment so not sure I can help. Maybe @Abhay Krishna Arunachalam can assist if available.

fad3t

2024-10-18 11:28:49

I'm having the same issue on vSphere by the way

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-18 11:29:19

Can you share your make target and user vars?

fad3t

2024-10-18 11:29:22

but I'm passing user data as well there

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-18 11:29:33

Ohhhh.... you're overriding the user data?

fad3t

2024-10-18 11:30:03

for vSphere yes, cause the original code uses ubuntu server ISO, while i'm using ubuntu cloud image (just like Nutanix)

fad3t

2024-10-18 11:31:25

I'll have quick lunch then share some more details

👍 Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-18 11:31:26

Oh, I'm not really sure how to handle that. Might need to wait for @Abhay Krishna Arunachalam unless there's someone else here that can help

Christophe Jauffret

2024-10-18 12:01:51

@Marcus Noble with last PR Nutanix build are ok on my side

fad3t

2024-10-18 12:42:29

so my make target is make build-nutanix-ubuntu-2204 but I'm setting PACKERVARFILES with 3 files: .json (Nutanix cluster info), .json (ubuntu22.04 stuff) and .json (K8s version)

fad3t

2024-10-18 12:43:07

I'll see if I can simply remove the user_data from my OS json file, it might have precedence over whats rendered by the template

Christophe Jauffret

2024-10-18 12:44:45

yes remove the user_data from your own file

fad3t

2024-10-18 12:46:12

I confirm it works by removing it from my custom variable file

👍:skin_tone_2: Christophe Jauffret

fad3t

2024-10-18 12:46:29

for vSphere I guess I'll have to maintain a patch file for it

fad3t

2024-10-18 13:26:44

I'm thinking about adding a make target for vSphere + Ubuntu cloud image, I'm just wondering whether people would be interested by this.

Abhay Krishna Arunachalam

2024-10-18 18:45:30

Hey sorry just seeing this. Do y'all still need me to look into something?

fad3t

2024-10-18 20:14:27

Nope, it's fixed - thx

🙌 Abhay Krishna Arunachalam

Brad P

2024-10-19 06:36:47

@Brad P has joined the channel

Brad P

2024-10-19 07:01:53

Hello everyone! I'm new to image builder and trying to get it to build an image using a local vmware workstation 17.6.1 install on Ubuntu 22.04 machine. I cloned the repo and attempted:

 make build-node-ova-local-ubuntu-2204 PACKER_LOG=1

but it always seems to get stuck here:

2024/10/19 05:45:05 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:05 [INFO] Attempting SSH connection to 172.16.175.136:22...
2024/10/19 05:45:05 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:05 [DEBUG] reconnecting to TCP connection for SSH
2024/10/19 05:45:05 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:05 [DEBUG] handshaking with SSH
2024/10/19 05:45:09 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:09 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/10/19 05:45:09 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:09 [DEBUG] Detected authentication error. Increasing handshake attempts.

I've noticed it seems to be setting a random password each time, and I attempt to login with the password that is set in packer-common.json, for example:

"ssh_password": "VhS9RszwfbP1idLQ",

I get the login prompt, but I can't login myself either, can't figure it out. Any help would be appreciated!

Abhay Krishna Arunachalam

2024-10-19 09:32:35

Which user are you using to login?

Brad P

2024-10-19 18:09:28

builder, it looks to be the user in packer-common.json

Brad P

2024-10-22 03:53:59

Hello, just following-up, this was a pretty basic setup, just minimal install of ubuntu with build essentials, jq, VMware workstation, etc, and the image builder. with all the defaults for the local OVA build, am I missing anything?

Thanks!

Slackbot

2024-10-21 15:30:17

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2024-10-21 15:30:43

I'll be on the train to London so won't be able to make this I'm afraid

👍 mboersma

mboersma

2024-10-21 15:30:54

No topics on the agenda currently, but if you have anything please add it!

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-21 15:50:07

I've just arrived at London so can make it if we have anything to discuss 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-21 16:16:35

Agenda still empty. What you wanna do @mboersma?

mboersma

2024-10-21 16:52:46

I guess we skipped! Sorry, was waiting for something to notify that there was an item.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-21 16:54:17

yeah no worries 🙂

hakman

2024-10-25 11:47:57

@hakman has joined the channel

Florian Löhden

2024-10-28 12:49:48

@Florian Löhden has joined the channel

Hani

2024-10-28 15:40:18

@Hani has joined the channel

Hani

2024-10-28 15:47:02

Hello team,
I am having a problem with AWS AMI images build with the latest release of image-builder.
The nodes seem to join the cluster without INTERNAL-IP, therefore calico doesn't run.
Any idea why this might be happening ?

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-28 15:51:05

As far as I'm aware we haven't had any changes recently that might effect that but I can't say for sure without more information.

What versions were you running before?

What Make target are you using?

What user vars are you providing (with anything sensitive removed)?

Hani

2024-10-29 08:14:50

I haven't used image-builder before

make build-ami-ubuntu-2204

these are the only vars i provided

{
    "aws_region": "eu-north-1",
    "ami_regions": "eu-north-1",
    "kubernetes_semver": "v1.30.5",
    "ami_groups": "",
    "snapshot_groups": "",
    "kubernetes_cni_semver": "v1.6.0"
}

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 08:20:33

Oh, so this isn't something thats broken in the latest release but more a case of it not working how you expected? Is that right?

Can you remind me what INTERNAL-IP is in this context? Is that a Kubernetes thing or an EC2 thing?

Hani

2024-10-29 08:23:55

I honestly don't know which case it is exactly.
Basically when you run kubectl get nodes -owide the nodes should have INTERNAL-IP showing up, but in my case it doesn't as if the kubelet doesn't get the address to advertise it

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 08:26:10

Oh gotcha
🤔 Trying to recall where that comes from. I've just checked in one of my own clusters (which uses Flatcar rather than Ubuntu) and I see the IP populated.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 08:26:35

Maybe someone who build Ubuntu AMIs can confirm if they see in the internal IP or not for nodes? 🙏

Hani

2024-10-29 08:31:43

I would add, that i tried building a Flatcar AMI, but there i run into another problem, the kubelet didn't manage to start reporting that no kubelet confing not found
This might be a very unlucky case of things not working for me somehow.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 08:33:45

I just checked my vars from back when we did build ubuntu (months back now) and I noticed that I have this var defined:

"kubernetes_cni_deb_version": "**"

but I don't have kubernetescnisemver. Are you needing that specific version of the cni?

Hani

2024-10-29 08:35:22

No necessarily, i was trying to have the latest versions of network related things, hoping it solves the problem

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 08:37:07

Can you try it without that set and see if that changes things? Would be good to rule it out at least.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 08:37:14

Maybe try setting "kubernetescnideb_version": "**" instead

richcase

2024-10-29 08:49:17

Do you get the same with one of the published AMIs?

Hani

2024-10-29 08:57:03

Yeah, i am having the same problem with published Ubuntu AMIs,

richcase

2024-10-29 08:59:41

CAPA uses calico in its e2e tests for non-EKS clusters.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 09:00:33

Actually, while I think about it - we are talking about non-EKS right?

Hani

2024-10-29 09:18:55

Yeah, non-EKS

Hani

2024-10-29 09:20:59

Building with "kubernetescnideb_version": "" fails the test with this

 {
    amazon-ebs.ubuntu-24.04:             "duration": 191859,
    amazon-ebs.ubuntu-24.04:             "err": null,
    amazon-ebs.ubuntu-24.04:             "expected": [
    amazon-ebs.ubuntu-24.04:                 ""
    amazon-ebs.ubuntu-24.04:             ],
    amazon-ebs.ubuntu-24.04:             "found": [
    amazon-ebs.ubuntu-24.04:                 "[\"1.4.0-1.1\"]"
    amazon-ebs.ubuntu-24.04:             ],
    amazon-ebs.ubuntu-24.04:             "human": "Expected\n    \u003c[]string | len:1, cap:1\u003e: [\"1.4.0-1.1\"]\nTo satisfy at least one of these matchers: [%!s(matchers.ContainElementMatcher=\u0026{0xc00040b4d0}) %!s(matchers.ContainElementMatcher=\u0026{0xc00040b560})]",
    amazon-ebs.ubuntu-24.04:             "meta": null,
    amazon-ebs.ubuntu-24.04:             "property": "version",
    amazon-ebs.ubuntu-24.04:             "resource-id": "kubernetes-cni",
    amazon-ebs.ubuntu-24.04:             "resource-type": "Package",
    amazon-ebs.ubuntu-24.04:             "result": 1,
    amazon-ebs.ubuntu-24.04:             "successful": false,
    amazon-ebs.ubuntu-24.04:             "summary-line": "Package: kubernetes-cni: version:\nExpected\n    \u003c[]string | len:1, cap:1\u003e: [\"1.4.0-1.1\"]\nTo satisfy at least one of these matchers: [%!s(matchers.ContainElementMatcher=\u0026{0xc00040b4d0}) %!s(**matchers.ContainElementMatcher=\u0026{0xc00040b560})]",
    amazon-ebs.ubuntu-24.04:             "test-type": 0,
    amazon-ebs.ubuntu-24.04:             "title": ""
    amazon-ebs.ubuntu-24.04:         }

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 09:41:33

Hmmm... maybe that var is no longer valid. As I say this was from months back now 😅

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 09:41:58

But, if the published AMIs are also not working for you I suspect something else is going on instead as I know people are successfully using those images.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 09:42:26

Do you specify any kubeadm config or similar when creating your cluster?

Hani

2024-10-29 09:53:53

nope, just following quickstart guide

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 09:54:58

What quickstart guide? The CAPA one?

Hani

2024-10-29 09:55:22

Yep, that one

Hani

2024-10-29 14:17:00

I can say now, that i managed to pin down the problem to the tigera-operator
thanks for the help

Marcus Noble (k8s@marcusnoble.co.uk)

2024-10-29 14:22:12

Oh interesting. Any idea why?

Hani

2024-10-30 08:27:09

I don't know exactly yet, but this is the error it, coreDNS etc.. give
Error from server: no preferred addresses found; known addresses: []

Hani

2024-10-30 15:56:43

The problem was related to this
which required the CCM to run in host network, i believe i took a long way around to figure out.
thanks for help!

GitHub

kubelet: cloud-provider external addresses by aojea · Pull Request #121028 · kubernetes/kubernetes

Kubelet, when using cloud provider external, initializes temporary the node addresses using the --node-ip flag values if set, until the cloud provider overrides it. This behavior has undesired cons...

Original URL: https://github.com/kubernetes/kubernetes/pull/121028

timothysc

2024-10-29 21:11:43

@timothysc has left the channel

orin

2024-11-01 10:59:21

Hi all, I have built Flatcar OS image (latest version from main branch on github) and load image into OpenStack Glance, and then I created a server from this image with my custom user data (I show it as yaml file, but I surely converted it to ignition format when booted my server)

variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: core
      password_hash: "$y$j9T$qRgyCaQq.RDwlXNoe.4lS1$srnHt2JI76LZIEQrk1wgMYGvedk/21f0LTWnzH9Z3uB"
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc2EAAAADAQA..........................xxxxx
    - name: stackops
      password_hash: "$y$j9T$qRgyCaQq.RDwlXNoe.4lS1$srnHt2JI76LZIEQrk1wgMYGvedk/21f0LTWnzH9Z3uB"
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc2EAA...............................xxxxxx
      shell: /bin/bash
      groups:
        - root
        - adm
        - wheel
        - sudo
        - systemd-journal
        - docker
storage:
  files:
    - path: /etc/ssh/sshd_config
      overwrite: true
      mode: 0600
      contents:
        inline: |
          UsePrivilegeSeparation sandbox
          Subsystem sftp internal-sftp
          UseDNS no

          PermitRootLogin no
          AllowUsers core
          AuthenticationMethods publickey

But when I tried to login my server using ssh core@x.x.x.x -i the server required me to using password to login, any can explain this bugs to me or give me a solution, I surely that chmod 400 my pem file. Thank you so much

tormath1

2024-11-04 08:44:32

Hi, if you have access to Horizon dashboard you might be able to access the instance console and check that your SSH keys are correctly injected under .ssh/authorizedkeys
Another question: why do you set the passwordhash for core user if the goal is to SSH with public keys only? if you enter the password does it work?
EDIT: You can now use this SSHd configuration:

storage:
  files:
    - path: /etc/ssh/sshd_config.d/custom.conf
      overwrite: true
      mode: 0600
      contents:
        inline: |
          # Do not allow root user
          AllowUsers core

Slackbot

2024-11-04 15:30:18

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2024-11-04 15:31:03

I'm not going to make it unfortunately.

mboersma

2024-11-04 16:58:30

I'm also not available, out of town this week.

hakman

2024-11-04 19:36:31

@hakman has left the channel

leland knight

2024-11-05 03:05:09

I'd like to run image-builder via a gitlab pipeline using a gitlab runner running via kubernetes, but image-builder is using a callback server ... Since I'm running via kubernetes, the local ip of the callback server can't be used, instead I have to expose the ip ... which I am able to do; but I'm not sure how to tell image-builder to use the exposed ip. How can I specify the ip to use?

fad3t

2024-11-05 07:18:15

never found a solution to this, instead I'm using Ubuntu cloud images, which are bootstrapped using cloud-init (no need to get the kickstart config from the Packer web server).

Sriraman Srinivasan

2024-11-05 11:03:36

will httpip , httpportmin and httpport_max do the trick ?

fad3t

2024-11-05 11:12:42

that's difficult, because he says Packer runs in a container so it gets a new (pod) IP every time, and it might not be accessible from outside the cluster (for the VM to reach it)

Sriraman Srinivasan

2024-11-05 11:13:08

Service IP?

fad3t

2024-11-05 11:13:40

what happens if you have 2 parallel runs of image builder?

Sriraman Srinivasan

2024-11-05 11:19:00

Just thinking out loud:
Create onfly service with selector pointing to the specific pod instance?

Sriraman Srinivasan

2024-11-05 11:22:00

Another option is if you use a git based solution, you could host init config files as a server say using nginx.

fad3t

2024-11-05 11:27:18

those config files are built by Packer and are dynamic

Sriraman Srinivasan

2024-11-05 11:30:57

those config files are built by Packer and are dynamic

I am talking about files under say example <a href='https://github.com/kubernetes-sigs/image-builder/tree/main/images/capi/packer/ova/linux'></a>
During the build, packer hosts these files inside http server it creates. Instead of then packer hosting it onfly, you host those seperately

GitHub

image-builder/images/capi/packer/ova/linux at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/tree/main/images/capi/packer/ova/linux

fad3t

2024-11-05 11:34:22

I see, this might work yeah

Sriraman Srinivasan

2024-11-05 11:35:42

you could use git based auto-update or something to make sure server is always uptodate

Travis Loyd

2024-11-05 17:04:33

via the gitlab runner helm chart it is possible to ask for a loadbalancer ip, or expose the runner via an ingress FQDN (which gets added to dns automatically), if there were just a place to specify that ... but image-builder tries to use the ClusterIP of the pod which is unreachable.

Travis Loyd

2024-11-05 17:09:26

I was starting to think about submitting a pull request, something like if 'such-and-such' env var existed, use that for the callback ip, otherwise detect the ip automatically ... but it was looking a lot harder than that. ... was hoping it wouldn't require a feature request; seems like this is going to become a more and more common use case

Travis Loyd

2024-11-05 17:11:13

it can still bind to the clusterip, it's just the thing calling back that needs to call the alternative specified fqdn or ip

Travis Loyd

2024-11-05 17:16:24

I'm not sure what is trying use the callback ip, when is the ip getting passed along which is being used to call back on? Its getting passed to the vm somehow to use after things are setup, if I wanted to submit a pull request, which git repo do I need to look at?

Travis Loyd

2024-11-05 17:16:51

we just want to pass an alternative fqdn or ip that we specify via env var

Travis Loyd

2024-11-05 17:37:22

@fad3d i submitted a feature request in the packer project, is that the right place? If so, maybe give it a thumbs up so I'm not alone:

leland knight

2024-11-15 00:48:32

I'm still trying to do this but having trouble figuring out where the code is that I'd have to change. Is the code different for every provider? So if I fixed it for proxmox it would only make proxmox builds work in kubernetes?

leland knight

2024-11-15 00:49:09

Is it in the provider, in imagebuilder, in packer, in the packer-sdk?

Sriraman Srinivasan

2024-11-15 03:20:40

Except the playbooks, majority is provider centric...

leland knight

2024-11-15 03:27:41

How could I influence things such that all providers might one day being able to run via kubernetes? There a kind of "standards group" I could suggest the idea to?

Sriraman Srinivasan

2024-11-15 03:33:34

At this stage of the prj, I guess this is going to be a little difficult since lots of properties specifically wrt packer is provider specific(packer plugins are differnet and hence different set of props required). Plus bootstrap/cloud-init differs again based on provider-OS combination with not all providers supporting all OS flavours....

Sriraman Srinivasan

2024-11-15 03:35:25

May be you can add agenda for this in doc and this then can be taken up during IB office hours

leland knight

2024-11-15 03:36:21

Being that I'm willing to put in the work, I kind of hate to let the idea die, but if it's just not a good idea I can let it go. I can just build images in a way that doesn't run in kubernetes. It's just that I prefer everything run 100% in k8s, making this one the first outlier for me.

👍 Sriraman Srinivasan

leland knight

2024-11-15 03:36:30

Thanks, I'll give that a shot

Travis Loyd

2024-11-05 17:00:17

@Travis Loyd has joined the channel

Travis Loyd

2024-11-05 17:20:12

I'm wanting to specify a different callback ip for the vm to use once it finishes. At what point in the code is the ip being passed to the vm? Is that in the 'image-builder' code?

I submitted a feature request in the packer project, is that the right place?

Gerhard Häring

2024-11-07 05:38:39

@Gerhard Häring has joined the channel

Andrew Sauber

2024-11-08 14:05:21

@Andrew Sauber has joined the channel

karine.santos

2024-11-12 14:41:21

hello everyone!

I use the openstack packer provider in creating my kubernetes images with image-builder. Is there a way for me to see the details of this image (services that go up together, etc.?)

Abhay Krishna Arunachalam

2024-11-12 22:23:56

Do you mean the output image on disk after image-builder has finished building it?

Tobias Thiel

2024-11-14 10:16:03

@Tobias Thiel has joined the channel

feitnomore

2024-11-14 16:29:36

I'm trying to build-qemu-ubuntu-2404, and I'm getting SSH error

feitnomore

2024-11-14 16:30:39

qemu: Setting up proxy adapter for Ansible....
==> qemu: Executing Ansible: ansible-playbook -e packerbuildname="qemu" -e packerbuildertype=qemu -e packerhttpaddr=10.0.2.2:8984 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerdurl= containerdsha256=041fa3cfd4e6689d37516e4c7752741df0974e7985d97258c1009b20f25f33c7 pauseimage=registry.k8s.io/pause:3.9 containerdadditionalsettings= containerdcrisocket=/var/run/containerd/containerd.sock containerdversion=1.7.20 containerdwasmshimsurl=--linux-x8664.tar.gz containerdwasmshimsversion=v0.11.1 containerdwasmshimssha256={"lunatic":"7054bc882db755ce5f3ded46d114bfd4e0a318e437fa18a2601295d20b616b32","slight":"a6ea87d965037933a7d9edb5e20cfc175265c8e1ca92a16535f1f3c3f376f5b0","spin":"dcffedb8e4d2f585a851b3de489fa1e8a0054ec0ad72cf111c623623919245d0","wws":"e917f90692d798d80873aa0f37990c7d652f2846129d64fecbfd41ffa77799b8"} containerdwasmshimsruntimes="" containerdwasmshimsruntimeversions="{"lunatic":"v1","slight":"v1","spin":"v2","wws":"v1"}" crictlurl= crictlsha256= crictlsourcetype=pkg customrolenames="" firstbootcustomrolespre="" firstbootcustomrolespost="" nodecustomrolespre="" nodecustomrolespost="" disablepublicrepos=false extradebs="qemu-guest-agent" extrarepos="" extrarpms="qemu-guest-agent" httpproxy= httpsproxy= kubeadmtemplate=etc/kubeadm.yml kubernetesapiserverport=6443 kubernetescnihttpsource= kubernetescnihttpchecksum=sha256: kubernetesgoarch=amd64 kuberneteshttpsource= kubernetescontainerregistry=registry.k8s.io kubernetesrpmrepo= kubernetesrpmgpgkey= kubernetesrpmgpgcheck=True kubernetesdebrepo= kubernetesdebgpgkey= kubernetescnidebversion= kubernetescnirpmversion= kubernetescnisemver=v1.2.0 kubernetescnisourcetype=pkg kubernetessemver=v1.29.10 kubernetessourcetype=pkg kubernetesloadadditionalimgs=false kubernetesdebversion=1.29.10-1.1 kubernetesrpmversion=1.29.10 noproxy= pipconffile= pythonpath= redhatepelrpm= epelrpmgpgkey= reenablepublicrepos=true removeextrarepos=false systemdprefix=/usr/lib/systemd sysusrprefix=/usr sysusrlocalprefix=/usr/local loadadditionalcomponents=false additionalregistryimages=false additionalregistryimageslist= ecrcredentialprovider=false additionalurlimages=false additionalurlimageslist= additionalexecutables=false additionalexecutableslist= additionalexecutablesdestinationpath= additionals3=false buildtarget=virt amazonssmagentrpm= enablecontainerdaudit= kubernetesenableautomaticresourcesizing= debugtools=false ubunturepo= ubuntusecurityrepo= gpublocknouveauloading= --extra-vars ansiblepythoninterpreter=/usr/bin/python3 --extra-vars --scp-extra-args "-O" -e ansiblesshprivatekeyfile=/tmp/ansible-key2403733080 -i /tmp/packer-provisioner-ansible683821705 /home/marcelo/local/src/github.com/feitnomore/image-builder/images/capi/ansible/node.yml
qemu:
qemu: PLAY [all] **
==> qemu: ssh: handshake failed: EOF

Slackbot

2024-11-18 15:30:14

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-11-18 15:58:54

Not sure I’ll make this. Currently in Copenhagen trying to find my Airbnb 😅

mboersma

2024-11-18 16:13:55

Nice, hopefully that's starting a vacation!

I put a couple things on the agenda just hopefully, but IDK if we actually have any updates. We just haven't met in a while and I wanted to touch base, but if we're too busy today I'll move them to the next slot.

Drew Hudson-Viles

2024-11-18 16:14:37

I won't be around for a few weeks afet today - happy to push if needed but just an fYI there 🙂

I don't have any updates on those items though

mboersma

2024-11-18 16:16:28

Sounds good. We can keep it short if no one has any updates or other topics.

Drew Hudson-Viles

2024-11-18 16:19:50

👍

Marcus Noble (k8s@marcusnoble.co.uk)

2024-11-18 16:24:29

Nice, hopefully that's starting a vacation!

Nope! Speaking at KCD Denmark tomorrow! 😁

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-11-18 16:25:30

Regarding the two topics on the agenda - there’s no update from my side. Tests are still in the same state I think and the Packer stuff isn’t moving and likely won’t.

mboersma

2024-11-18 16:41:39

Drew and I chatted about topics, but we don't really have any updates yet. I'll carry them over to the next meeting and maybe we've thought of something relevant by then. 🙂

Yongxiang Gao

2024-11-20 03:15:09

I try to use image-builder to build ubuntu 22.04, someone knows how to use USG (Ubuntu Security Guide) to harden the image in image-builder?

I try to use "usg fix cislevel1server" in ansible task, however, when I use the generated osImage to provision EKSA bare metal cluster, I see such error:

"failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"

"failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"

Which tailor rule can be used to skip cilium and kubelet during usg fix?

rajas

2024-11-26 11:08:02

Hey folks! 👋 just created based on the conversation I had with @Marcus Noble at Kubecon!

The idea is to see if there’s appetite for this in the community and work on a proposal together!

GitHub

Golang implementation of image-builder with buildkit backend · Issue #1633 · kubernetes-sigs/image-builder

Is your feature request related to a problem? Please describe. Currently, there are bunch of issues with the build and test of image-builder There is no clear demarkation between OS level tasks and...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1633

:parrotk8s: Marcus Noble, Abhay Krishna Arunachalam, Sriraman Srinivasan, galop

❤️ bavarianbidi, chrischdi

👍 tormath1

Marcus Noble (k8s@marcusnoble.co.uk)

2024-11-26 11:17:31

I've also pinned this issue in the repo to try and get more visibility and feedback on it 🙂

:thank_you: rajas, Sriraman Srinivasan

Kepler SysAdmin

2024-11-28 12:35:27

Hi Community,
I've been using the image-builder for about 6-9 months now,
most of the time, it worked correctly, and if I encounter an issue, it was because I was missing something on the var_file.json

Nevertheless, since yesterday, there is something that is quite unique when it tries to build the image.
On the step Add the Kubernetes repo key is failing:

    openstack: TASK [kubernetes : Add the Kubernetes repo key] *
    openstack: fatal: [default]: FAILED! => {"after": ["D94AA3F0EFE21092", "871920D1991BC93C"], "before": ["D94AA3F0EFE21092", "871920D1991BC93C"], "changed": true, "fp": "234654DA9A296436", "id": "234654DA9A296436", "key_id": "234654DA9A296436", "msg": "apt-key did not return an error, but failed to add the key (check that the id is correct and *not a subkey)", "short_id": "9A296436"}

I think this issue is quite recent and maybe a few people are affected.
The json file I am using:

{
  "source_image": "",
  "networks": "",
  "flavor": "gp1.small",
  "floating_ip_network": "public",
  "image_name": "ubuntu-2204-kube-v1.27.16",
  "image_visibility": "public",
  "image_disk_format": "raw",
  "volume_size": "20",
  "volume_type": "",
  "ssh_username": "ubuntu",
  "kubernetes_deb_version": "1.27.16-1.1",
  "kubernetes_semver": "v1.27.16",
  "kubernetes_series": "v1.27"
}

I'll appreciate your help.
Thank you!!!

Kepler SysAdmin

2024-11-28 13:24:18

by the way, such task is:

---
- name: Add the Kubernetes repo key
  ansible.builtin.apt_key:
    url: "{{ kubernetes_deb_gpg_key }}"
    state: present

Where:

"kubernetes_deb_gpg_key": " user kubernetes&#95;series }}/deb/Release.key"
"kubernetes_series": "v1.27"

Hardie Boeve

2024-11-28 17:57:25

@Hardie Boeve has joined the channel

Piyush Maru

2024-11-29 05:52:54

@Piyush Maru has joined the channel

Razvan Mihai

2024-11-29 09:25:29

@Razvan Mihai has joined the channel

Kentaro Abe

2024-12-01 14:28:45

@Kentaro Abe has joined the channel

Slackbot

2024-12-02 15:30:30

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2024-12-02 15:38:07

I bumped the topics from last time to this one, but I'm not sure we have any updates. But I'm happy to get together even if it's brief.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-12-02 15:41:02

Yeah no updates as far as I know but it'd be good to meet up as I suspect this'll be that last of the year anyway.

👍 mboersma, rajas

rajas

2024-12-02 16:02:36

I am not able to see this meeting on the kubernetes calendar -

what's the best way to add this meeting to my calendar. Usually joining the mailing list works but that seems to be not working in this case

Google Workspace

Google Calendar - Easier Time Management, Appointments & Scheduling

Learn how Google Calendar helps you stay on top of your plans - at home, at work and everywhere in between.

Original URL: https://calendar.google.com/calendar/u/0/embed?src=calendar@kubernetes.io

mboersma

2024-12-02 16:22:06

Yes, and we're not able to get it on the calendar. It's properly set up in the community repo, but the tooling simply fails to publish several sig-cluster-lifecycle events, and image-builder is one of them.
SCL volunteers have put a fair amount of time into trying to fix this or find a workaround and eventually gave up. Sorry, not a good answer I know.

rajas

2024-12-02 16:25:03

but the context helps. Thanks! I'll setup up a reminder myself for now

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2024-12-03 11:55:04

Image-builder v0.1.40 is now available:
Thanks to all contributors! 🎉

GitHub

Release v0.1.40 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.40">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.40</a> What's Changed Update docs for v0.1.39 by @AverageMarcus in ...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.40

🎉 mboersma, rajas

:thank_you: mboersma, rajas

Kentaro Abe

2024-12-03 16:34:22

Hello community.
I'm trying to build ubuntu-2404-qemu images using VM hosted in Proxmox.
but It couldn't.

first, It takes so long time to connect ssh and those message shown continually. (for about 20 minutes)

2024/12/02 18:45:19 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:19 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:19 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:19 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:26 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:26 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:26 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:26 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:26 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:26 [DEBUG] handshaking with SSH
2024/12/02 18:45:28 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:28 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:28 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:28 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:35 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:35 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:35 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:35 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:35 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:35 [DEBUG] handshaking with SSH
2024/12/02 18:45:38 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:38 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:38 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:38 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:45 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:45 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:45 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:45 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:45 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:45 [DEBUG] handshaking with SSH
2024/12/02 18:45:48 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:48 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:48 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:48 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:55 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:55 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:55 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:55 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:55 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:55 [DEBUG] handshaking with SSH
2024/12/02 18:45:58 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:58 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:58 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:58 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:46:05 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:46:05 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:46:05 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:46:05 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:46:05 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:46:05 [DEBUG] handshaking with SSH

after success to connect, SSH session disconnect unexpectedly.

==> qemu: Provisioning with Ansible...
    qemu: Setting up proxy adapter for Ansible....
2024/12/02 18:54:00 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:00 Creating inventory file for Ansible run...
2024/12/02 18:54:00 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:00 SSH proxy: serving on 127.0.0.1:40695
==> qemu: Executing Ansible: ansible-playbook -e packer_build_name="qemu" -e packer_builder_type=qemu -e packer_http_addr=10.0.2.2:8167 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=041fa3cfd4e6689d37516e4c7752741df0974e7985d97258c1009b20f25f33c7 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.7.20 containerd_wasm_shims_url=--linux-x86_64.tar.gz containerd_wasm_shims_version=v0.11.1 containerd_wasm_shims_sha256={"lunatic":"7054bc882db755ce5f3ded46d114bfd4e0a318e437fa18a2601295d20b616b32","slight":"a6ea87d965037933a7d9edb5e20cfc175265c8e1ca92a16535f1f3c3f376f5b0","spin":"dcffedb8e4d2f585a851b3de489fa1e8a0054ec0ad72cf111c623623919245d0","wws":"e917f90692d798d80873aa0f37990c7d652f2846129d64fecbfd41ffa77799b8"} containerd_wasm_shims_runtimes="" containerd_wasm_shims_runtime_versions="{"lunatic":"v1","slight":"v1","spin":"v2","wws":"v1"}" crictl_url= crictl_sha256= crictl_source_type=pkg custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_apiserver_port=6443 kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_goarch=amd64 kubernetes_http_source= kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key= kubernetes_rpm_gpg_check=True kubernetes_deb_repo= kubernetes_deb_gpg_key= kubernetes_cni_deb_version= kubernetes_cni_rpm_version= kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=pkg kubernetes_semver=v1.30.5 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.30.5-1.1 kubernetes_rpm_version=1.30.5 no_proxy= pip_conf_file= python_path= redhat_epel_rpm= epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= ecr_credential_provider=false additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= additional_s3=false build_target=virt amazon_ssm_agent_rpm= enable_containerd_audit= kubernetes_enable_automatic_resource_sizing= debug_tools=false ubuntu_repo= ubuntu_security_repo= gpu_block_nouveau_loading= --extra-vars ansible_python_interpreter=/usr/bin/python3 --extra-vars  --scp-extra-args "-O" -e ansible_ssh_private_key_file=/tmp/ansible-key1609096398 -i /tmp/packer-provisioner-ansible138304401 /home/systemadmin/image-builder/images/capi/ansible/node.yml
    qemu:
    qemu: PLAY [all] **
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 SSH proxy: accepted connection
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 authentication attempt from 127.0.0.1:39488 to 127.0.0.1:40695 as builder using none
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 authentication attempt from 127.0.0.1:39488 to 127.0.0.1:40695 as builder using publickey
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 new env request: LANG=C.UTF-8
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 new exec request: /bin/sh -c '( umask 77 && mkdir -p "echo /tmp/.ansible"&& mkdir "echo /tmp/.ansible/ansible-tmp-1733165641.684557-2704-253976952207637" && echo ansible-tmp-1733165641.684557-2704-253976952207637="echo /tmp/.ansible/ansible-tmp-1733165641.684557-2704-253976952207637" ) && sleep 0'
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [DEBUG] Opening new ssh session
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [ERROR] ssh session open error: 'EOF', attempting reconnect
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [DEBUG] handshaking with SSH
2024/12/02 18:54:01 [INFO] 0 bytes written for 'stdin'
==> qemu: ssh: handshake failed: EOF
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [INFO] 0 bytes written for 'stdout'
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [INFO] 0 bytes written for 'stderr'
2024/12/02 18:54:01 [INFO] 0 bytes written for 'stderr'
2024/12/02 18:54:01 [INFO] 0 bytes written for 'stdout'
Read from remote host 172.16.223.13: Connection reset by peer
Connection to 172.16.223.13 closed.
client_loop: send disconnect: Broken pipe

Do you have any solutions for this problems?

leland knight

2024-12-08 01:23:40

does the vm have a screen like this?

image.png

leland knight

2024-12-08 01:29:34

If so, there should be a line something like this:

2024/12/08 01:27:09 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/12/08 01:27:09 Found available port: 8395 on IP: 0.0.0.0
==> proxmox-iso.ubuntu-2204: Starting HTTP server on port 8395

Which is an HTTP server setup that the vm calls back to in order to load its configuration. This may be that the vm can't reach back to the HTTP server. To help understand you can test that HTTP server like this (the port may vary):

$ curl localhost:8395

22.04/
24.04/

Kiran

2024-12-04 01:52:34

@Kiran has joined the channel

Kiran

2024-12-04 02:24:29

👋 Hello, team!

I am trying to add a cronjob inside a distroless container that is running the application. It is either giving error or not running correctly. How can get this working as I want to create a cronjob that runs a shell script & a python script.

This is the Dockerfile I'm using:

 # Base image for building the application
 FROM 
 docker.io/debian:12-slim AS build
 
 # Set Python environment variables
 ENV PYTHONDONTWRITEBYTECODE 1
 ENV PYTHONUNBUFFERED 1
 
 # Create directory for Gunicorn logs
 RUN mkdir -p /app/logs/gunicorn
 
 # Install necessary dependencies and libraries
 RUN apt-get update && <br>     apt-get install --no-install-suggests --no-install-recommends --yes <br>     python3-venv python3-dev default-libmysqlclient-dev build-essential <br>     libmariadb-dev pkg-config wget curl gnupg2 unzip cron <br>     && apt-get clean <br>     && rm -rf /var/lib/apt/lists/**
 
 # Set up Python virtual environment
 RUN python3 -m venv /pypi/venv && <br>     /pypi/venv/bin/pip install --upgrade pip setuptools wheel
 
 # Copy crontab
 COPY web/crontab /etc/cron.d/crontab
 RUN chmod 0644 /etc/cron.d/crontab
 RUN touch /var/log/cron.log
 
 # Install Python dependencies
 FROM build AS build-venv
 COPY web/requirements.txt /requirements.txt
 RUN /pypi/venv/bin/pip install --disable-pip-version-check -r /requirements.txt pymysql wfastcgi gunicorn gevent
 
 # Final stage: Set up the runtime environment
 FROM gcr.io/distroless/python3-debian11
 
 # Copy necessary files from previous stages
 COPY --from=build-venv /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
 COPY --from=build-venv /pypi/venv /pypi/venv
 COPY --from=build-venv /app/logs /app/logs
 COPY --from=build /etc/cron.d/crontab /etc/cron.d/crontab
 COPY --from=build /var/log/cron.log /var/log/cron.log
 
 # Set environment variables
 ENV PYTHONPATH=web:$PYTHONPATH
 
 # Copy application code
 COPY . /app
 WORKDIR /app
 
 
 # Start Gunicorn
 ENTRYPOINT ["cron", "&&", "/pypi/venv/bin/gunicorn", "web.APP.wsgi:application", "--bind", "0.0.0.0:8000", "--access-logfile", "/app/logs/gunicorn/access.log", "--error-logfile", "/app/logs/gunicorn/error.log", "--log-level", "info"]

Also tried using a entrypoint.sh script:

 #!/bin/sh
 
 # Start gunicorn in background
 /pypi/venv/bin/gunicorn web.APP.wsgi:application --bind 0.0.0.0:8000 --access-logfile /app/logs/gunicorn/access.log --error-logfile /app/logs/gunicorn/error.log --log-level info  &
 
 # Run periodic task
 while true; do
     python3 web/cron_script.py
     sleep 300  # Run every 5 minutes
 done

But with this also, I don't see the cron_script running.

Marcus Noble (k8s@marcusnoble.co.uk)

2024-12-04 06:18:20

I think you might be asking in the wrong channel. This channel is for the https://github.com/kubernetes-sigs/image-builder project.

GitHub

GitHub - kubernetes-sigs/image-builder: Tools for building Kubernetes disk images

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder

👍 Sriraman Srinivasan

Abhay Krishna Arunachalam

2024-12-05 08:40:22

Hello image-builder maintainers, I have a fix for some RHEL image build issues which I observed in our CI, and it could potentially happen to anyone installing Ansible collection community.general version >= v10.0.0

GitHub

Fix state for Ansible RHSM repository module by abhay-krishna · Pull Request #1646 · kubernetes-sigs/image-builder

I observed this in all the RedHat image builds in our CI qemu: TASK [sysprep : Remove subscriptions] ****************************************** qemu: fatal: [default]: FAILED! => {&quot;changed&...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1646

Sriraman Srinivasan

2024-12-05 09:05:07

@Marcus Noble / @mboersma PR submitted coresponding to issue. If either of you get time, can you please have a look at it.

GitHub

Enable cleanup for /var/log/messages on Photon for vSphere Provider by sriramandev · Pull Request #1648 · kubernetes-sigs/image-builder

Change description This PR adds logrotate binary and corresponding configurations to enable log rotation and cleanup for /var/log/messages in Photon for vSphere Provider Related issues Fixes # En...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1648

GitHub

Enable cleanup for /var/log/messages on Photon for vSphere Provider · Issue #1647 · kubernetes-sigs/image-builder

Enable cleanup for /var/log/messages on Photon for vSphere Provider What steps did you take and what happened? Create cluster and let it run for few days. Run applications which pushes output to /v...

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1647

Sriraman Srinivasan

2024-12-05 10:27:19

I have marked it as draft... wanted to make a small edit to variablize(not sure if its actually a word) the value for maxsize ...

Sriraman Srinivasan

2024-12-05 19:58:43

@mboersma I have marked it as ready for review and squashed the changeset.

👍 mboersma

Sriraman Srinivasan

2024-12-10 09:15:00

@mboersma Have addressed the comments..

:lgtm: mboersma

Sriraman Srinivasan

2024-12-12 09:45:32

Thanks @mboersma

Sriraman Srinivasan

2024-12-05 09:05:42

Whats the procedure to cut tag/make a new release? What are the conditions we do that(one I am assuming is when CAPI release or K8s release happens)

mboersma

2024-12-05 16:35:21

The release process is described here:

There isn't a release cadence currently, it's more based on when maintainers think changes in main justify tagging it. We just cut v0.1.140 a couple days ago.

leland knight

2024-12-08 01:51:57

Trying to use 'ISO_FILE' w/ proxmox provider:

ISO_FILE="tower:iso/ubuntu-22.04.5-live-server-amd64.iso"

Seeing error:

** one of iso_file, iso_url, or a combination of cd_files and cd_content must be specified for boot_iso

leland knight

2024-12-08 01:52:30

I added it to my proxmox.env file which I'm calling using this script:

$ cat go.sh 
#!/bin/bash

docker run -it --rm --net=host --env-file proxmox.env <br>  -v /tmp:/home/imagebuilder/images/capi/downloaded_iso_path <br>  registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.40 build-proxmox-ubuntu-2204

Franz

2024-12-10 17:32:48

@Franz has joined the channel

beddari

2024-12-12 09:38:10

@beddari has joined the channel

Rafael Polanco

2024-12-12 21:37:56

@Rafael Polanco has joined the channel

Franz

2024-12-13 13:33:17

Heya, I've been trying to build capi-images for a while now (for all of ubuntu-24.04, ubuntu-22.04, rocky9), but keep getting stuck at the same point -- both with qemu and with the remote-image builder on openstack. In all cases things die with a strange ansible error:

make build-qemu-ubuntu-2404 PACKER_LOG=1
....
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] 0 bytes written for 'stdout'
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] 0 bytes written for 'stderr'
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] RPC client: Communicator ended with: 0
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] 0 bytes written for 'stdin'
    qemu:
    qemu: TASK [Gathering Facts] *
    qemu: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /root/.ansible/tmp/ansible-local-1793709qosm0_m_/tmp5fp2y_ef /tmp/.ansible/ansible-tmp-1734093819.7338624-1793720-280401030368575/AnsiballZ_setup.py:\n\n"}
    qemu:
    qemu: PLAY RECAP *
    qemu: default                    : ok=1    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
    qemu:
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 shutting down the SSH proxy
2024/12/13 13:43:40 [INFO] (telemetry) ending ansible
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
2024/12/13 13:43:40 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 failed to unlock port lockfile: close tcp 127.0.0.1:5998: use of closed network connection
2024/12/13 13:43:40 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 failed to unlock port lockfile: close tcp 127.0.0.1:3020: use of closed network connection
2024/12/13 13:43:40 [INFO] (telemetry) ending qemu
==> Wait completed after 11 minutes 49 seconds
2024/12/13 13:43:40 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/12/13 13:43:40 machine readable: qemu,error []string{"Error executing Ansible: Non-zero exit status: exit status 2"}
==> Builds finished but no artifacts were created.
2024/12/13 13:43:40 [INFO] (telemetry) Finalizing.
==> qemu: Deleting output directory...
Build 'qemu' errored after 11 minutes 49 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 11 minutes 49 seconds

==> Some builds didn't complete successfully and had errors:
--> qemu: Error executing Ansible: Non-zero exit status: exit status 2

I'd appreciate any pointers on how to fix this.

Drew Hudson-Viles

2024-12-13 13:39:26

are there any other errors that stand out? Unfortunately the failed to unlock port lockfile: close tcp 127.0.0.1:5998: use of closed network connection is usually just a red herring as it's something else that's caused a failure and then this gets output as a result.

I can confirm openstack builds are working with the latest release as I ran one yesterday afternoon that was successful. QEMU should be too but I've not personally tested.

Franz

2024-12-13 13:44:58

I don't see anything else that might cause issues. However, if I run with FOREGROUND=1 I get the following:

make build-qemu-ubuntu-2404 PACKER_LOG=1 FOREGROUND=1
....
==> qemu: Starting VM, booting from CD-ROM
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Qemu version: 8.2.0
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Qemu Builder has no floppy files, not attaching a floppy.
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Executing /root/.local/bin/qemu-system-x86_64: []string{"-display", "gtk", "-vnc", "127.0.0.1:63", "-drive", "if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2", "-drive", "file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom", "-machine", "type=pc,accel=kvm", "-smp", "1", "-cpu", "host", "-device", "virtio-scsi-pci,id=scsi0", "-device", "scsi-hd,bus=scsi0.0,drive=drive0", "-device", "virtio-net,netdev=user.0", "-boot", "once=d", "-netdev", "user,id=user.0,hostfwd=tcp::4092_:22", "-m", "2048M", "-name", "ubuntu-2404-kube-v1.30.5"}
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Started Qemu. Pid: 1795078
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Qemu stderr: qemu-system-x86_64: -display gtk: Parameter 'type' does not accept value 'gtk'
==> qemu: Error launching VM: Qemu failed to start. Please run with PACKER_LOG=1 to get more info.
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 failed to unlock port lockfile: close tcp 127.0.0.1:5963: use of closed network connection
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 failed to unlock port lockfile: close tcp 127.0.0.1:4092: use of closed network connection
==> qemu: Deleting output directory...
Build 'qemu' errored after 15 seconds 720 milliseconds: Build was halted.

==> Wait completed after 15 seconds 720 milliseconds

==> Some builds didn't complete successfully and had errors:
--> qemu: Build was halted.

==> Builds finished but no artifacts were created.
2024/12/13 14:41:54 [INFO] (telemetry) ending qemu
==> Wait completed after 15 seconds 720 milliseconds
2024/12/13 14:41:54 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/12/13 14:41:54 machine readable: qemu,error []string{"Build was halted."}
==> Builds finished but no artifacts were created.
2024/12/13 14:41:54 [INFO] (telemetry) Finalizing.
2024/12/13 14:41:55 waiting for all plugin processes to complete...
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/YaleUniversity/goss/packer-plugin-goss_v3.2.13_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/hashicorp/qemu/packer-plugin-qemu_v1.1.0_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
make: ** [Makefile:543: build-qemu-ubuntu-2404] Error 1

Franz

2024-12-13 13:46:11

Don't know if that qemu-system-x86_64: -display gtk: Parameter 'type' does not accept value 'gtk' is causing things to die also without FOREGOUND=1 ?

Franz

2024-12-13 13:51:40

For the openstack remote build I also get

PACKER_VAR_FILES=openstack_vars.json make build-openstack-ubuntu-2204
....
    openstack:
    openstack: PLAY [all] *
    openstack:
    openstack: TASK [Gathering Facts] 
    openstack: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /root/.ansible/tmp/ansible-local-1726741dszour5u/tmpj3mgwet2 /tmp/.ansible/ansible-tmp-1733848994.5385716-1726748-172419909510439/AnsiballZ_setup.py:\n\n"}
    openstack:
    openstack: PLAY RECAP *
    openstack: default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
    openstack:
==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...

No other errors besides this one.

Drew Hudson-Viles

2024-12-13 14:00:01

ok cool. Well I'm running the qemu one now and it's working fine for me.

I presume you're building on linux?

Are you using the latest version of image builder?

Have you run make deps-xxxx for the appropriate build?

With qemu, can you manually launch a qemu instance outside of packer using the binary packer is using? Have a look for the Executing xxxxxx line in the log output when running with PACKERLOG=1 for an example.

With OpenStack - does PACKERLOG=1 give any extra information?

They are definitely working though so something locally is causing an issue here -

Franz

2024-12-13 14:57:07

Sorry for being slow in replying -- Christmas activities 🎅 🙂

yes linux; it's a VM that runs rocky9

I recently pulled the image-builder repo, I'm at commit ace9c5f8f

yes I've run both make deps-qemu and make deps-openstack

there are actually two executing lines:

Executing qemu-img: []string{"create", "-f", "qcow2", "output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5", "20480M"}

Executing /root/.local/bin/qemu-system-x8664: []string{"-smp", "1", "-machine", "type=pc,accel=kvm", "-netdev", "user,id=user.0,hostfwd=tcp::3020:22", "-drive", "if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2" "-drive", "file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom", "-boot", "once=d", "-m", "2048M", "-vnc", "127.0.0.1:98", "-cpu", "host", "-device", "virtio-scsi-pci,id=scsi0", "-device", "scsi-hd,bus=scsi0.0,drive=drive0", "-device", "virtio-net,netdev=user.0", "-name", "ubuntu-2404-kube-v1.30.5"}

I'm assuming you'd like me to run the latter? Or both?

no extra info with OpenstStack and PACKER_LOG=1

I agree, it has to be something with my local environment...

Franz

2024-12-13 15:13:43

I now ran both commands:

qemu-img create -f qcow2 output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5 20480M
Formatting 'output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16

and

root/.local/bin/qemu-system-x86_64 -smp 1 -machine type=pc,accel=kvm -netdev user,id=user.0,hostfwd=tcp::3020:22 -drive if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2 -drive file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom -boot once=d -m 2048M -vnc 127.0.0.1:98 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -name ubuntu-2404-kube-v1.30.5
(diskImageBuilder-venv) [root@cirrus-deploy capi]# /root/.local/bin/qemu-system-x86_64 -smp 1 -machine type=pc,accel=kvm -netdev user,id=user.0,hostfwd=tcp::3020:22 -drive if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2 -drive file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom -boot once=d -m 2048M -vnc 127.0.0.1:98 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -name ubuntu-2404-kube-v1.30.5                                                                                                                                                                                   
qemu-system-x86_64: warning: Machine type 'pc-i440fx-rhel7.6.0' is deprecated: machine types for previous major releases are deprecated

Now it just sits there. I guess that's a good thing?

Franz

2024-12-13 15:14:10

Gotta run for now. I'll pick this up again on Monday. Thanks a lot for looking into this!!!

Drew Hudson-Viles

2024-12-13 15:17:00

No problem. Yeah once you've run that, it means it's sitting there and the instance should be running. You can use a VNC client to try and connect to ensure it's working. This rules out any issue with the qemu binary you're using anyway if it is working and you can connect.
Have a good weekend and enjoy the festivities!

mboersma

2024-12-16 14:46:17

I have a "soft" conflict with office hours this moring, but I don't see anyting on the agenda, so I'm assuming we will skip. Hopefully I'm not wrong, please follow up here in Slack if there are questions.

Drew Hudson-Viles

2024-12-16 14:47:18

Yes I've got nothing today so I'm happy to skip this one. Not feeling too great either so more rest time is welcomed!

❤️ mboersma

Slackbot

2024-12-16 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

LEI

2024-12-16 21:09:04

@LEI has joined the channel

Richard Cunningham

2024-12-22 10:00:11

@Richard Cunningham has joined the channel

Richard Cunningham

2024-12-22 10:02:08

Hi. I am having issues building a qemu Ubuntu 24.04 image (22.04 works ok). The first task for ansible-playbook node.yaml fails to connect SSH. It looks like this issue; . Does anyone have suggestions how to debug further?

GitHub

Error Building Ubuntu 24.04 (QEMU) · Issue #1631 · kubernetes-sigs/image-builder

Environment Make target: make build-qemu-ubuntu-2404 Run using container image? (Y/N): N Environment vars: export PACKER_FLAGS="--var 'kubernetes_rpm_version=1.31.2' --var 'kubernetes_semver=v1.31....

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1631

Slackbot

2024-12-30 15:30:23

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2024-12-30 15:37:08

There's nothing currently on the agenda. Let me know if you have anything you'd like to discuss, otherwise we'll skip until next time (13 January 2025).

Sriraman Srinivasan

2024-12-31 13:40:19

New year wished to all and family....

🎉 Drew Hudson-Viles

zakaria

2025-01-03 16:12:38

@zakaria has joined the channel

zakaria

2025-01-03 20:35:48

Hello, we are trying to build a kubernetes VM image from an existing ova template on vSphere, we use the image builder project with the vspher-clone packer builder. At the cloning stage we get the following error knowing that the template folder exist and have in its full path a whitespace. We suspect that the builder does not correctly parse the fullpath ? can anyone help us on this matter? Thank you

image.png

Drew Hudson-Viles

2025-01-06 08:30:18

Hi,

I'm not a user of the vSphere side of things so I'm providing complete guesswork from my side.
It does seem like it's expected that no whitespace would be in the path though. The only things I can recommend is to either remove the whitespace from the path if possible or try supplying an escaped value for the space and see if that works.

If not, maybe someone who uses vSphere in anger can supply another option.

👍 zakaria

zakaria

2025-01-06 09:26:57

it seems that the vsphere plugin handles the path join, so we should have only provided the folder name directly not the fullpath 😅.

image.png

Drew Hudson-Viles

2025-01-06 09:28:08

Aaah ok! Well glad you sorted it in the end!

👍 zakaria

zakaria

2025-01-06 09:28:43

thank you 😃

Jennifer Weir

2025-01-04 22:07:35

@Jennifer Weir has joined the channel

Benjamin

2025-01-06 08:21:45

@Benjamin has joined the channel

Yongxiang Gao

2025-01-08 05:11:41

Quick question, why there is no preseed-efi.cfg for ubuntu 22.04?
We can see such for ubuntu 20.04 which use base file instead:

./packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
./packer/raw/linux/ubuntu/http/20.04/preseed-efi.cfg

I don't see much packages in packer/raw/linux/ubuntu/http/22.04/user-data, however, the image (tgz file) size is around 2GB, how to reduce it significantly?

Abhay Krishna Arunachalam

2025-01-08 09:36:30

The gz file, when decompressed, yields an image of size 6442450944 bytes which is exactly equal to 6144 MB , the disk size specified here in the raw packer.json. So I think by reducing the size here, you should be able to reduce the final image size. Although I must say I'm not sure about the side effects of doing so.

Yongxiang Gao

2025-01-10 01:51:34

Thanks, I can try, however, I tried to increase that value to 7G for some reason(s) I don't recall.
From a different way, are there packages can be excluded/removed some how to save image size to under 1GB?

s3rj1k

2025-01-13 17:14:49

preseed was deprecated a long time ago, not sure if it even works on newer Ubuntu versions

:ty: Yongxiang Gao

fad3t

2025-01-08 12:57:23

Hi everyone, I'm facing a weird issue and I'm running out of ideas; I'm hoping somebody from this channel would know 😛
I'm building Ubuntu 22.04 images on Nutanix using image-builder, and for some reasons I would like to use a specific "release" of the Ubuntu 22.04 cloud image -- with a specific kernel version/ABI. I've tried setting the image_url to an older release and deployed the resulting template on a cluster, but the kernel version reported doesn't match the one from the Ubuntu release. I've tested with multiple release, they all seem to have kernel version 5.15.0-130-generic which is probably the latest for 22.04.
I'm trying to figure out if the problem comes from the upstream image(s), from the infrastructure building the image or image-builder itself (is it maybe updating the kernel during the build?). Any hint or ideas would be welcome.

Christophe Jauffret

2025-01-08 13:54:23

there is a apt dist-upgrade in the image builder process, i imagine this what replace your kernel version

hajowieland

2025-01-08 21:30:13

@hajowieland has joined the channel

Slackbot

2025-01-13 15:30:29

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-13 15:39:26

☝️ anything to discuss today? Agenda currently empty

mboersma

2025-01-13 16:09:36

Nothing in particular from me. I guess we can punt.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-13 16:12:00

👍 Skipping...

s3rj1k

2025-01-13 16:31:55

hmm, was hoping to jump in 😞

s3rj1k

2025-01-13 16:32:08

ok, so can we do quick offline maybe?

s3rj1k

2025-01-13 16:32:14

hi, all 🙂

mboersma

2025-01-13 16:32:54

I can fire up the zoom meeting just-in-time, no worries.

s3rj1k

2025-01-13 16:33:10

I am there 🙂

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-13 16:52:02

Damn, sorry just seen this. I’m not at my laptop now

mboersma

2025-01-13 16:57:54

No worries @Marcus Noble, we had a good discussion and we'll summarize in the document.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-13 17:04:05

Thanks

s3rj1k

2025-01-13 16:31:21

@s3rj1k has joined the channel

s3rj1k

2025-01-13 17:08:12

Hi all, I was wondering is there someone here that uses raw target to build images? especially Ubuntu and/or Flatcar images

s3rj1k

2025-01-13 17:08:29

cc: @mboersma

👍 mboersma

Abhay Krishna Arunachalam

2025-01-13 22:34:03

Hello, yes we do use image-builder to build Ubuntu 20.04/22.04 and RHEL 8/9 Raw images. Image-builder currently doesn't have Ubuntu 22.04 raw image support so we patch it to include that functionality. The Ubuntu 22.04 subiquity autoinstall files for qemu just worked for raw builds as well and we validate these images in our e2e tests as well

s3rj1k

2025-01-13 23:45:43

Whoa, nice one, please consider doing a pr

mboersma

2025-01-14 16:13:29

There is also this related PR: It would be great if someone could carry that through to merge.

s3rj1k

2025-01-14 19:20:48

^^ I can test raw (Ubuntu) target manually, if some one has the ability to test other changes would be nice, or we can split PR maybe?

s3rj1k

2025-01-14 19:25:29

asked original author in PR about this

s3rj1k

2025-01-15 19:58:21

I've created a consolidated PR for this,
Please take a look

s3rj1k

2025-01-16 11:01:08

@mboersma ^^ can you take a look please?

s3rj1k

2025-01-18 11:44:06

tests seem to pass

s3rj1k

2025-01-23 15:19:11

so are we able to land that change?

zakaria

2025-01-20 08:43:20

Hello all, I am using the build-node-ova-vsphere-clone-ubuntu-2204 to build a kubernetes v1.30.7 vSphere ova from the base template cluster-api-provider-vsphere 1.30.0. I am trying to build it without exposoing an HTTP server to serve the meta-data and user-data for the cloud-init, instead I tweeked the packer-node configuration for the vsphere-clone builder to use the cdfiles and cdlabel in order to pass them to the VM in an An iso CD. Unfortunatly, the build process get stuck on the Waiting fot ssh to be available step with the following msg, I suspect that the cloud-init does not take into consideration the user-data file? Any input on this issue would be appriciated.

image.png

zakaria

2025-01-20 08:46:09

I have added the following to the user-data config apt:
preservesourceslist: false
primary:
- arches: [default]
uri:

chrischdi

2025-01-20 14:05:34

I’d take a look to the vm packer creates maybe you get some clue out of that.

zakaria

2025-01-21 08:18:26

when I connect to the VM using the vSphere web console, I get the login prompt but no creds work for me, I have tried to connect with the builder user and its corresponding password (that I have changed in the set-ssh-paasword script so it does not generate one randomly)

zakaria

2025-01-21 08:19:05

i tried login: ubuntu and passwd ubuntu also it does not work

chrischdi

2025-01-21 09:09:14

Maybe you get some hint when watching the screen while it comes up

Sriraman Srinivasan

2025-01-22 06:08:42

@zakaria Depending on the commit you are running from, if older, builder/builder should work if I am not mistaken. In case of newer commit, check the console(script) to get the password(password get generated randomly during start and gets logged).

Sriraman Srinivasan

2025-01-22 06:13:48

Also if you are able to see that the VM has booted correctly with login prompt, check if the IP assigned to the VM and the one packer is waiting on are the same. If different, increase the ipsettletimeout to say 20m or something depending upon the the time taken to boot up of the VM. Sometimes when bootup takes time (usually due to slower network - primarily to connect to Ubuntu servers to do update), in such cases DHCP re-assigns a new IP and packer will not know of the new IP and waits on the older IP. ipsettletimeout will cause packer to wait for the mentioned time before trying to get the IP of the VM

zakaria

2025-01-22 08:16:32

@Sriraman Srinivasan I am using a static IP, with the linux customize options ( It seems that image builder project does not support it by default, I had to add the customize config to the packer-node.json file), and I set the ssh_host to the same ip address.

zakaria

2025-01-22 08:18:56

I am using the latest image builder version v0.1.40 that uses the packer version v1.9.5

zakaria

2025-01-22 08:22:15

I have hardcoded the $ENCRYPTEDSSHPASSWORD to have a simple password in the set-ssh-password script, As I understand in the user-data config, it sources the builder user password from this variable.

zakaria

2025-01-22 08:22:35

but it does not work 😅

zakaria

2025-01-22 08:24:00

I suspect that the cloud-init does not take into consideration the user-data I have configured

zakaria

2025-01-22 08:25:33

I have set the user-data as a VM property, also does not work

2025-01-20 16:27:44

@SM has joined the channel

leland knight

2025-01-23 22:06:41

A humble request:

Though we all use image-builder to create the images for our kubernetes clusters we are currently unable to use it within kubernetes pipeline solutions. So, if you use an on-prem solution such a gitlab or jenkins, running in kubernetes, you will not be able to run image-builder.

This is because the hashicorp packer project has a limitation which breaks via kubernetes. I've created a fix, which requires three tiny changes in three git repos. I humbly request, if you would like to run image-builder within a kubernetes environment that you give my pull request over there a thumbs up. I'm not sure the folks reviewing my pull request have enough kubernetes experience to really understand how freeing using their tool via kubernetes would be.

Thank you for your consideration.

Additionally, I began working on a strategy to test everything image-builder related, but my solutions are always kubernetes based meaning I can't pursue those goals without being able to run things within a kubernetes environment. ()

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 06:51:15

Why are you unable to run in Kubernetes exactly? I run image-builder in Kubernetes with Tekton pipelines successfully for CAPA, CAPZ and CAPV

fad3t

2025-01-24 07:33:36

I'm also running image-builder within Kubernetes, with CAPX and CAPV.
For CAPV I have to apply a patch to the Makefile because we're using Ubuntu cloud images (so vApps) but overall its OK.

leland knight

2025-01-24 15:29:46

because when the vm is complete it sends an http message back to packer, the ip it uses is the ip of the pod, the clusterip, and so the vm fails to signal that its finished

leland knight

2025-01-24 15:30:34

if you've already got a solution for that i'm all ears, i've put a ton of effort into this and would love to know there's already a method to make it work

fad3t

2025-01-24 15:32:01

If i'm not mistaken the HTTP call is used to get the "kickstart" (not sure about the term here). So if you're using cloud-init, this is not needed as all the information needed to bootstrap the VM is already known (via userdata).

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:33:39

What make target are you calling?

leland knight

2025-01-24 15:34:42

i'm essentially running this same command but wanting to do it via kubernetes rather than on my vm with docker:

docker run -it --rm --net=host --env-file proxmox.env <br>  -v /tmp:/home/imagebuilder/images/capi/downloaded_iso_path <br>  registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.38 build-proxmox-ubuntu-2204

()

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:36:58

🤔 Maybe this is something Proxmox-specific then?

leland knight

2025-01-24 15:38:27

I have tried to solve the issue in a way that would be easy for any provider to implement. Logically, each provider could implement their own unique non-standard work around, which I suspect the more popular providers have probably done.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:41:05

Do you know what exactly in the VM performs the callback? I didn't think Packer actually ran on the VMs when building but I could be wrong about that.

leland knight

2025-01-24 15:43:36

there's a place in the proxmox provider where it binds to 0.0.0.0 the http server the vm will talk with, it then runs a function that tries to get the ip address of the system, then provides the ip address it finds to the vm; my solution adds a new variable to the packer sdk, then the proxmox provider checks if that variable exists, if it does it uses that as the ip address instead of running the function to get the ip of the system id did the 0.0.0.0 bind with; there is another variable which looks it can be used for that purpose but its slightly different in that it offers an alterative bind address when it exists instead of 0.0.0.0

fad3t

2025-01-24 15:46:16

it this the HTTP server mentioned here then?

GitHub

image-builder/images/capi/packer/proxmox/ubuntu-2204.json at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/proxmox/ubuntu-2204.json#L2

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:48:12

Ah ok, so this is a Proxmox feature. That explains why no one else has reported it. We have very few Proxmox users.
It's this functionality, right?

GitHub

packer-plugin-proxmox/builder/proxmox/common/step_type_boot_command.go at main · hashicorp/packer-plugin-proxmox

Packer plugin for Proxmox Builder. Contribute to hashicorp/packer-plugin-proxmox development by creating an account on GitHub.

Original URL: https://github.com/hashicorp/packer-plugin-proxmox/blob/main/builder/proxmox/common/step_type_boot_command.go

leland knight

2025-01-24 15:50:31

I don't think its necessarily proxmox specific; Its been awhile, but when I was deep diving this a few months ago I think what I found was some sample code that people could use when they wanted to put together a provider, and that sample code has this problem (feature?) in it. So I'd say its more sample-code related, and probably in multiple providers, than specifically proxmox.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:51:04

Sorry, let me say "proxmox-specific in image-builder"

leland knight

2025-01-24 15:51:08

though at the moment i'd have to dive in again, i'm not sure if it was clusterapi sample code or image builder sample code

fad3t

2025-01-24 15:52:18

i'm not sure this is specific to Proxmox, is it? when using ubuntu 22 live server ISO, it expects the boot command, which usually refers to an HTTP server (which is hosted by the Packer host)

leland knight

2025-01-24 15:53:19

one sec, reviewing my own pull requests which point out where it is in the code

leland knight

2025-01-24 15:54:28

here's the proxmox specific change, but i suspect the same if statement is in all the providers, i believe this is sample code ... i'll look for the sample code so i can show you what i mean ...

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:54:29

Ok I guess what I really want to know then is why this isn't impacting the other providers in the same way then. If we can figure that out then maybe we can solve it for Proxmos

fad3t

2025-01-24 15:55:51

for CAPV I end up with the same issue if I'm using the Ubuntu live server ISO

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:56:30

Oh really?!

fad3t

2025-01-24 15:56:35

because the packer pod has a non routable (pod) IP, which is set by Packer in the boot command - but the VM cannot reach it

👍 leland knight

fad3t

2025-01-24 15:57:04

might work if using a routable pod IP, never had a chance to test this

leland knight

2025-01-24 15:57:50

I suspect if we ran image-builder via kubernetes with every provider and each of the images many would build the image, timeout after like 20 minutes, and fail because the vm couldn't report in.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:57:56

ok - so the reason I've never hit this in our environment then is we build Flatcar not Ubuntu which I guess doesn't have this behaviour because of ignition.

👍 leland knight

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:58:48

But how are we not having lots of people complaining about this? 😕 Surely this would be a blocker for plenty of people?

fad3t

2025-01-24 15:59:09

I remember seeing a couple of issues from people failing to build Ubuntu on CAPV

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 15:59:38

OH! Is that this same issue?

leland knight

2025-01-24 16:01:13

maybe when it doesn't work people just give up and switch to another solution, or they aren't running pipelines in k8s cause they just haven't got to that point in their experience with kubernetes, or they just use aws and maybe that one works (i haven't tried), or when it doesn't work they just give up and run docker via a vm and hope someone else fixes the problem some day

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:01:18

This issue?

GitHub

build-qemu-ubuntu-2204 stuck in "Waiting for SSH to become available..." · Issue #1076 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1076

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:01:44

You might be right 😞

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:02:55

I'm guessing then that ssh doesn't become available because the VM didn't initialise and setup ssh server because it couldn't reach back to packer

👍 leland knight

leland knight

2025-01-24 16:03:13

that looks like it, "not planned", yet my pull request would solve that one ... so i should link it to my pull request

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:04:01

"not planned" is because it went stale as no-one could solve it or offer more info

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:04:15

(I hate stale bots)

leland knight

2025-01-24 16:04:15

which is understandable, i worked on it for weeks before figuring it out

fad3t

2025-01-24 16:04:19

so your PR would allow to override the HTTP server IP right? how would you determine the IP to use then?

leland knight

2025-01-24 16:05:05

it took a long time cause i had to learn code in three repos to solve it, and well open source so didn't have a ton of time ... but i became a little obsessed i'll admit

😆 Marcus Noble

leland knight

2025-01-24 16:05:27

@fad3t we'd just have to set it via an environment variable

leland knight

2025-01-24 16:05:51

in my case i use gitlab which uses a gitlab-runner in kubernetes and it makes the ip available as an env var

fad3t

2025-01-24 16:06:35

but what IP would you set? cause the runner has a dynamic IP, every time it runs it gets a different IP

leland knight

2025-01-24 16:07:32

exactly

leland knight

2025-01-24 16:07:59

the helm chart has a setting 'use loadbalancer' or just clusterip, so i set that to loadbalancer

fad3t

2025-01-24 16:08:14

I see, thx

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:08:38

Hold on, before we go too far with this - we have tests in place on image-builder that successfully build ubuntu-2204 on Azure. Maybe we should have a look at what that provider does differently.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:08:49

E.g. https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_image-bui[…]/1877240240804990976/artifacts/azure-vhds/ubuntu-2204.log

leland knight

2025-01-24 16:10:15

sorry, i am supposed to be at work ... so i have to come back to this when my day ends, i can look into that log, but i can look into their custom solution after work, and can work on things this weekend

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:10:53

Oh I'm not saying you need to solve it, or we need to solve it right now. I'm just saying its something to look at as an alternative possible solution 🙂

leland knight

2025-01-24 16:11:06

i'm definitely motivated to get this implemented, and my preference is to solve it for all providers if possible, rather than rewrite proxmox and let the issue remain elsewhere

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:11:45

Sure, but if we can solve it without needing a load balancer then that would be my personal preference 🙂

leland knight

2025-01-24 16:11:58

since you both understand the issue, if you could help to communicate it over in that packer-sdk pull request, that'd be helpful to help out the person making the decision to include it or not

leland knight

2025-01-24 16:12:56

@Marcus Noble i suppose there's nothing stopping us from putting in the kubernetes specific url, in my case something like gitlab-runner.gitlab.svc so we can use the clusterip

leland knight

2025-01-24 16:13:19

but we'd still need the implementation i've put together (as far as i can tell)

leland knight

2025-01-24 16:14:05

maybe if you could both just add a note over there that the issue does exist, and that the pull requests could solve it, though i understand folks might decide as a group there's a better way

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:14:07

What I mean is the Azure provider manages without the need of calling back to a http server it seems. I'd like to understand how it does that.

leland knight

2025-01-24 16:15:04

maybe that could be an action item for you?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:15:09

Ah! I think Azure uses Azure Resource Manager which gives it more capabilities

👍 leland knight

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:15:14

Yeah, I'm looking

👍 leland knight

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:16:20

My concern mainly is even if we get that PR in, how do we handle this in environments where the cluster isn't routable from the cloud (that's the case for me). Also, what security implications are there in opening up that endpoint to the web? (As I don't understand this enough)

leland knight

2025-01-24 16:17:22

it could just be like pod.cluster.svc

leland knight

2025-01-24 16:17:30

it doesn't have to be a loadbalancer

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:17:35

We also have GCP in our tests building Ubuntu20.04 -

leland knight

2025-01-24 16:17:42

i'd use an ingress for sure, but it'd all just be internal in the end

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:18:35

Yeah, that'd work in your specific case but I'm trying to think about the other environments this may run. And also thinking about the recent security incidents we've had and make sure we don't introduce anything more. 🙂

leland knight

2025-01-24 16:18:51

dang it, i want to work on this, but have to go to actual job ... afk for awhile, will be in and out

👍 Marcus Noble

leland knight

2025-01-24 16:18:56

ya, let's solve it for all providers

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:19:01

Just to be clear - I'm not trying to block anything here, I just want to figure out all the options 🙂

leland knight

2025-01-24 16:19:12

so everyone isn't implementing something unique, unique solutions make fore security vulnerabilities

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:19:31

Yeah, but I suspect that is totally what's happening 😞

leland knight

2025-01-24 16:19:37

me too

fad3t

2025-01-24 16:19:37

Azure and GCE seem to use pre-existing images (kind of AMIs) so they probably don't need the boot command

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:20:05

Ah! Good point! I guess that'll be the same for AWS too then.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:22:44

I need to head off also as it's end of my day now. I'm going to try and think through this over the weekend. If we go down this route I think we need some automated way to expose the endpoint securely but I'm not sure we will be able to as there's nothing that says image-builder is running in a cluster. 🤔

fad3t

2025-01-24 16:23:57

at the end its just a way to override the IP address that is announced to the VM, its not changing the way it is actually running

fad3t

2025-01-24 16:24:58

if there's a security risk linked to this, then it is already present as packer is already running on that image builder k8s pod

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-24 16:25:02

Yeah, I’m just trying to figure out how this will work in image-builder

Drew Hudson-Viles

2025-01-24 21:44:17

I've just read all of this thinking "but I don't have this problem in OpenStack either" - then I read the last 4 comments and the penny dropped. The problem here is systems that don't use the boot command are generally fine. Anything that does evidently seems to be having this issue.

That's a fascinating bug though and whilst it appears to be an edge case where it errors out, it's not an unlikely scenario that would cause it IE someone wanting to actually build something in K8S which relies on a boot command.

Not sure what the solution is yet, just wrapping my head around it all myself and thought I'd chime in with my 2 cents!

👍 leland knight, Marcus Noble, mboersma

chrischdi

2025-01-26 10:33:01

For vsphere/capv afaik it also works in a mode where the cloud init data is provided via an iso instead of relying on the http callback.

chrischdi

2025-01-26 10:38:14

Especially remembering this part in that pr https://github.com/kubernetes-sigs/image-builder/pull/1459/files#diff-92590d094272c3e33208dbb6a15aff2a1f1e9d866b7411a2c38271e57f5b2728R78

chrischdi

2025-01-26 10:39:20

Because upstream ci (when it still existed) did run packer in a pod inside the prow cluster. There was no way to call back from the VM to the pod where packer was running.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:08:13

Any idea how that is used @chrischdi? Looks like we install it into the container image so maybe we can use that for Proxmox then? I dunno what it actually does though.

chrischdi

2025-01-27 09:09:28

One question is: does the proxmox plugin allow to mount a second iso to get the cloud-init data from.

chrischdi

2025-01-27 09:09:48

and allow to upload the built iso (so it can be mounted). But I never used proxmox.

chrischdi

2025-01-27 09:10:02

(I guess it allows to do so but can’t confirm)

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:10:40

You can upload ISO to proxmox, that bit I know. I don't know about the first. Can you show me how it's used in vsphere provider and I'll see if I can figure it out

chrischdi

2025-01-27 09:11:56

I think its a default packer feature.
For OVA builds if I’m right these are the important lines for it:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:13:11

Ah cool, and then its used like here:

chrischdi

2025-01-27 09:13:39

ack yeah that makes sense

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:13:53

ok, let me investigate then 🙂

chrischdi

2025-01-27 09:14:06

(the cd_label is also important for this if I remember correctly)

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:14:19

cidata ?

chrischdi

2025-01-27 09:14:28

jep

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:15:11

🤔

chrischdi

2025-01-27 09:15:15

even documented for proxmox:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:17:37

ah yes! It's about half way down the page (annoyingly no anchor to link to)

Screenshot 2025-01-27 at 9.17.30 AM.png

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:19:49

So it might be possible to add:

  "cd_content_location": "./packer/proxmox/linux/{{user distro&#95;name}}/http/{{user distro&#95;version}}/**",
  "cd_label": "cidata",

to the proxmox ubuntu values?

chrischdi

2025-01-27 09:22:59

could be it, yeah

chrischdi

2025-01-27 09:23:45

IMHO the most robust way for this stuff as the network back to packer is not needed.

:nod: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 09:24:12

Until this thread I didn't even know the http call was how it worked 🙈

chrischdi

2025-01-27 09:39:06

😄 well its not how it always works 🙂

chrischdi

2025-01-27 09:39:18

but afaik is the default when using packer

leland knight

2025-02-13 21:14:46

If no one has come up with a better way, maybe consider the solution I've put together and comment on the pull requests / github issues.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-14 06:55:32

Have you tried the cd approach? That would make it inline with others and not require HTTP connectivity.

leland knight

2025-02-14 17:38:04

I'm not sure what you mean by cd, so no probably. I'll scroll up in this chat and look for 'cd'.

leland knight

2025-02-14 17:41:44

Still, I mean ... even if there is a workaround, why not solve it so a work-around isn't required.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-14 18:09:53

It isn’t a workaround. It’s the approach used elsewhere in the project.

leland knight

2025-02-14 18:10:31

I feel resistance to the idea I've proposed, I don't want to go against the flow, could you help me to see why the idea I've suggested isn't a good idea?

leland knight

2025-02-14 18:11:02

I'm just curious, genuinely interested in your insight before I give up ... I mean, the hard part has already been done ... I've already implemented a solution.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-14 18:26:48

I’m not suggesting that. I’m just asking to try the approach that we already have used with other providers and see if that solves the problem. That way we don’t need any external changes.

leland knight

2025-02-14 18:28:47

k, sorry, what do you mean by cd? that translates as 'continous delivery' in my brain

leland knight

2025-02-14 18:32:37

this cd?

image.png

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-14 19:35:14

Yeah that. Sorry I’m on my phone. I should have got you a link.

👍 leland knight

leland knight

2025-02-14 19:35:58

no worries, working on several other things at the moment ... appreciate the response

👍 Marcus Noble

Victor Sartori

2025-01-27 11:44:41

@Victor Sartori has joined the channel

Slackbot

2025-01-27 15:30:03

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 15:31:32

☝️ Anyone want to use the office hours to discuss the above thread? (Or is the thread good enough for now?) Any other topics as the agenda is currently empty?

Drew Hudson-Viles

2025-01-27 15:41:28

I'm easy on this - I think the thread is ok for the moment unless we've some immediate action we want to take?

Drew Hudson-Viles

2025-01-27 15:42:00

I do have to leave to pick my daughter up at 4:45 though so I'd have to do a quick one for me 😄

mboersma

2025-01-27 15:44:54

I don't have any agenda items in particular, but happy to talk. I just now caught up with the megathread.

mboersma

2025-01-27 15:45:17

(Was out on Friday last week)

Drew Hudson-Viles

2025-01-27 15:46:12

tbf, It has been a while so at the very least we should have a catch up 🙂

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 15:46:42

Works for me 🙂

s3rj1k

2025-01-27 16:07:56

I have a question in regards to PR, so it would be nice to have some short meetup

Marcus Noble (k8s@marcusnoble.co.uk)

2025-01-27 16:11:28

I'm having some internet issues so I might not actually be able to join. Go ahead without me if I'm not there

👍 Drew Hudson-Viles

Andy Townsend

2025-01-28 20:46:30

@Andy Townsend has left the channel

Ilya Alekseyev

2025-02-09 23:08:58

@Ilya Alekseyev has joined the channel

laozc

2025-02-10 07:41:06

@laozc has joined the channel

Danila B.

2025-02-10 13:27:24

@Danila B. has joined the channel

Slackbot

2025-02-10 15:30:02

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2025-02-10 15:30:51

Unfortunately I won't be able to make this one. We've a new starter today and I'm in meetings until around 5:30 UK time.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-10 15:32:04

I'm also not going to make it today but the agenda is looking empty so I think it's ok to skip.

👍 Drew Hudson-Viles

mboersma

2025-02-10 15:33:04

Works for me, see you next time!

icelynjennings

2025-02-18 20:36:25

@icelynjennings has joined the channel

Vitaliy Kozlovskiy

2025-02-19 18:19:28

@Vitaliy Kozlovskiy has joined the channel

Amulyam24

2025-02-20 06:47:10

@Amulyam24 has joined the channel

Priyanshi Khetwani

2025-02-20 06:47:21

@Priyanshi Khetwani has joined the channel

Priyanshi Khetwani

2025-02-20 07:08:04

Hi Team,
I am trying to build an image using below command

PACKER_LOG=1 PACKER_FLAGS="--var 'kubernetes_rpm_version=1.30.4-0' --var 'kubernetes_semver=v1.30.4' --var 'kubernetes_series=v1.30' --var 'kubernetes_deb_version=1.30.4-00'" make build-qemu-ubuntu-2204

but getting ssh handshake error

Attempting SSH connection to 127.0.0.1:2877...
reconnecting to TCP connection for SSH
handshaking with SSH
SSH handshake err: ssh: handshake failed: read tcp 127.0.0.1:33368->127.0.0.1:2877: read: connection reset by peer

Also if I try without some packer variables, i am able to build the image

PACKER_LOG=1 PACKER_FLAGS="--var 'kubernetes_semver=v1.30.4'" make build-qemu-ubuntu-2204

Can someone please help?

cc: @Amulyam24

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-20 15:35:22

I think you might be hitting this issue (which closed without a resolution by the looks of it 😞 )

GitHub

build-qemu-ubuntu-2204 stuck in "Waiting for SSH to become available..." · Issue #1076 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/issues/1076

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-20 15:35:48

Does anyone have a process for adding custom tags to AWS AMIs when building them with image-builder?

Slackbot

2025-02-24 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-24 15:31:32

I had totally forgot about this 🤦‍♂️ I'm going to need to skip. Got too much that still needs to get done.

mboersma

2025-02-24 15:33:00

No worries Marcus! I can be there but the agenda is currently empty, so unless someone speaks up soon I think we will skip until next time.

:thx_thanks: Marcus Noble

Abhay Krishna Arunachalam

2025-02-24 18:49:07

Hello @Marcus Noble @mboersma @Drew Hudson-Viles just curious, what is the tentative date for the next IB release v0.1.41?

mboersma

2025-02-24 18:51:36

The project doesn't have a release schedule, we've been doing releases when "enough" new features or bug fixes have landed. We may be overdue for one.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-02-24 18:54:09

Ah! I knew there was something I wanted to mention 🤦‍♂️ Yeah. I think we're overdue one, I noticed we have a few unreleased merges the other day.

mboersma

2025-02-24 18:54:33

There's a PR in the queue right now that may merge, but after that I can do a release.

💙 Marcus Noble, Abhay Krishna Arunachalam

👍 Drew Hudson-Viles

Abhay Krishna Arunachalam

2025-02-24 20:40:08

Thank you!

mboersma

2025-02-24 21:42:37

Thank you for the nudge!

🙌 Abhay Krishna Arunachalam

Abhay Krishna Arunachalam

2025-02-24 21:43:54

Especially excited for this one because a couple of IB patches we're maintaining on EKS-anywhere side have been upstreamed (thanks to @s3rj1k 🎉)

❤️ mboersma, s3rj1k, Marcus Noble

mboersma

2025-02-25 15:52:48

kubernetes Image-builder v0.1.41 is now available:
Thanks to all contributors!

🙌 Drew Hudson-Viles, Victor Sartori, Abhay Krishna Arunachalam, s3rj1k

mohamed karim

2025-02-25 16:36:28

@mohamed karim has joined the channel

mkumatag

2025-02-26 00:37:58

@mkumatag has left the channel

2025-02-27 02:05:42

@Mo has joined the channel

Razvan Mihai

2025-02-27 09:57:40

@Razvan Mihai has left the channel

Ayan Khan

2025-03-03 11:14:59

@Ayan Khan has joined the channel

Ayan Khan

2025-03-03 11:56:29

How can we know the password of the default user "ubuntu". Or is there any option to configure it as I'm not able to find anything related to this. Found in a discussion that a random UUID is taken for password during image build process.

Drew Hudson-Viles

2025-03-03 11:59:37

It depends on the provider you're using.

For example, the QEMU provider has it dynamically set via a script which generates a packer.json from this template:

Other builders will allow you to pass it in as a var - you'll have to look at which provider you are using and the builder it invokes to pass the correct value through.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-03 12:00:03

The UUID password is for the "builder" user that should only be used during the initial image creation. I thought you're asking for an ssh user within the generated image at the end, yes?

Drew Hudson-Viles

2025-03-03 12:00:41

aaah yeah sorry, I may have misread that Marcus - good catch about the resulting image 😄

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-03 12:01:27

I don't thing, by default, we generate any ssh user in the resulting image. (I could be wrong, not sure about all providers and OS's)

Drew Hudson-Viles

2025-03-03 12:02:22

To my knowledge no we do not. The method I mentioned is about setting one during the build process.

Generally speaking you would pass in things like credentials/user creation or public keys for your ssh keypair via cloud-init .

:gr_correct: Sriraman Srinivasan

Ayan Khan

2025-03-03 12:16:25

Thanks everyone! Really appreciate the help.

Dave Miles

2025-03-05 10:55:11

@Dave Miles has joined the channel

Victor Sartori

2025-03-05 16:42:09

Hey everyone,

I recently opened a PR to add support for a new provider, Canonical MaaS.
What’s the process for adding a new provider? Should this be discussed with someone in particular, or would it be a good topic for the next office hours meeting?
Here’s the PR:

GitHub

Add support for Canonical MaaS Provider by vasartori · Pull Request #1695 · kubernetes-sigs/image-builder

Change description Add support for Canonica MaaS Related issues Fixes Canonical MaaS as an Image Provider #1688 Additional context Initially, this version supports only Ubuntu 22.04 and Ubuntu ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1695

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-06 11:24:08

Hey Victor, I'm sorry I meant to respond to your PR and just lost track of time with everything going on.
I'm having a chat with the other maintainers about this (and related) as we're at a tipping point where we're not sure if we can commit to managing any more providers or operating systems. We think we might need to come up with some sort of minimum set of requirements for adding new ones but not sure yet what that might be. Ideally, any new providers would also come with tests but that's really only viable if the providers themself are willing to donate infrastructure to run tests on (do you know if this is something Canonical is likely to be interested in?)
I think it makes sense to talk about this in the office hours.

/cc @mboersma @jsturtevant @Drew Hudson-Viles

Victor Sartori

2025-03-06 11:31:59

Hello Marcus,
I'm not sure if Canonical is interested in this. By the way, I don't work at Canonical but at another company that is integrating with MaaS.
Regarding donating infrastructure to run MaaS, I can check internally at my company. It might be feasible, but I need to discuss it with many people to make it happen.
IMHO, I think this would be a very interesting and important step for the CAPI project to support bare metal directly. As you know, with the rise of AI and related technologies, the use of bare metal will increase considerably, and MaaS makes it much easier to deploy a Kubernetes cluster.

Victor Sartori

2025-03-06 11:33:37

The next office hour meeting will be Mon, 10th march?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-06 11:37:08

Correct

Victor Sartori

2025-03-06 11:37:51

and can I add this topic to the agenda?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-06 11:40:52

Yeah go right ahead 🙂

Victor Sartori

2025-03-06 11:41:02

🙂

Victor Sartori

2025-03-14 11:02:59

Hello @Marcus Noble
Sorry to bother you with this, but I'd like to go over some points we discussed in the last office hours.
I left a comment on the PR , asking if my understanding of our discussion was correct.
I'm not sure if I fully understood which README (or if this should perhaps go into the book?) and what its content should be. If you could help with some topics, I can continue from there.
I'm asking because I want to speed up the process of getting this provider merged into master as much as possible, so we can move forward with our project more smoothly.

GitHub

Add support for Canonical MaaS Provider by vasartori · Pull Request #1695 · kubernetes-sigs/image-builder

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1695

Victor Sartori

2025-03-14 11:05:55

And regarding the owners file, who is responsible for updating it? Should I do it? How does this process work?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-14 11:08:48

I’m currently travelling, @Drew Hudson-Viles or @mboersma are you available to help out?

Victor Sartori

2025-03-14 11:09:56

Oh, Marcus, I'm really sorry to bother you with this!! Enjoy your trip!

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-14 11:11:27

It’s totally fine 🙂

Drew Hudson-Viles

2025-03-14 11:49:51

We're just launching a (quiet) go live today so might have limited time but I will try and find some if I can and if Matt doesn't get in there before me!

Victor Sartori

2025-03-14 11:51:28

Got it! no worries if it’s not possible today. Good luck with the go-live!

👍 Drew Hudson-Viles

Yongxiang Gao

2025-03-06 09:36:22

Does image-builder support full disk encryption (boot disk) for ubuntu 22.04 now?

Drew Hudson-Viles

2025-03-06 21:13:30

Hi, responded in the DM you sent 🙂

:ty: Yongxiang Gao

hemaa

2025-03-08 13:30:25

@hemaa has joined the channel

Thomas Güttler

2025-03-10 08:45:08

Why chronyd, when provider has matching NTP set?

I compare the result of make build-hcloud-ubuntu-2404 with a vanilla ubuntu24.04 created in hcloud.

The vanilla image has this setting:

/etc/systemd/timesyncd.conf.d/hetzner.conf:NTP=ntp.hetzner.com

I think this works fine.

Why does image-builder install chronyd?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 08:52:56

Looks like originally it was added for Azure I think?

Not sure if it's a case of copy from one provider to another.

Thomas Güttler

2025-03-10 12:38:08

thank you for the reply. I see that there is a lot of chrony config in image-builder. I will see if I can disable it.

chrischdi

2025-03-10 12:38:29

no background on ntp in image-builder, from personal experience in the past: chronyd behaves way better and I/we had experience in the past when using timesyncd

mboersma

2025-03-10 15:17:17

Apparently our automated reminder expired, but image-builder office hours start in 15 minutes. Agenda:

👍 Drew Hudson-Viles

:thx_thanks: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 15:20:50

Daylight savings time change 😄

mboersma

2025-03-10 15:21:03

D'oh of course.

mboersma

2025-03-10 15:22:02

So the time switched here, but not yet in the UK? So is ~10 minutes from now the right time, or am I just making US-based assumptions?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 15:22:24

The UK switched ages ago. We're back on UTC 😄

mboersma

2025-03-10 15:22:29

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 15:22:39

My notification popped up so I think you're right

mboersma

2025-03-10 15:22:53

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 15:22:54

As we schedule it based on US time

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 15:23:49

Maybe I should delete this and let you recreate? Then it should (hopefully) maintain the right time with the US?

Screenshot 2025-03-10 at 3.23.28 PM.png

mboersma

2025-03-10 15:24:40

Sure, we can try that, maybe that's the way.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-10 15:24:55

Deleted 🙂

mboersma

2025-03-10 15:29:36

Ok, I recreated it for 8:30 am my time. Had to fiddle around to get it to start two weeks from now.

💙 Marcus Noble

mboersma

2025-03-10 15:26:48

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 9:30AM every other Monday (next occurrence is March 17th), Mountain Daylight Time.

mboersma

2025-03-10 15:27:44

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 9:30AM every other Monday (next occurrence is today), Mountain Daylight Time.

mboersma

2025-03-10 15:29:11

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 8:30AM every other Monday (next occurrence is March 24th), Mountain Daylight Time.

Slackbot

2025-03-10 15:30:16

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Abhay Krishna Arunachalam

2025-03-11 06:13:15

Hello image-builder maintainers, I was trying to build GPU-ready OVAs by setting the vsphere ISO builder's pcipassthroughalloweddevice field. vSphere allows users to assign multiple PCI passthrough devices (GPU cards, video capture cards, audio cards, etc) to a virtual machine without specifying an exact physical device on a particular ESXi host. It does this through a feature called Dynamic DirectPath I/O, which requires virtual hardware version 17. However since we're hardcoding this to 15, I'm not able to get this feature working. This version hasn't been changed in 5 years, I think we should update this outdated version or remove it. From the Packer documentation, the vsphere builder's vmversion field, if not set, defaults to the most current virtual machine hardware version supported by the ESXi host. Kindly let me know your thoughts on this.

Abhay Krishna Arunachalam

2025-03-11 06:52:10

We have overriden the hardware version to 18 for windows 2019 and windows 2022, but only because the corresponding guest OS types were not supported in hardware version 15. Maybe we should bump the default to a newer version while still allowing overriding for backward compatibilty with older vCenter versions?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-11 06:58:22

It sounds like the unset default might actually be best if I understand correctly. That would then always use the latest version available in the environment you’re building the image, yes?
We should still allow it to be set if needed for backward compatibility though.

Abhay Krishna Arunachalam

2025-03-11 07:01:58

I haven't tried the unset default route myself and for the time being, am resorting to overriding vmx_version in the image-specific packer file (rhel-8 OVA.json for example), but based on the documentation, it seems like it should work

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-11 07:04:55

If theres no objections from others I'd be in favour of that. But I don't have all that much knowledge of OVA so maybe worth someone else weighing in first.

chrischdi

2025-03-11 20:48:43

In case you are using capv: you should be able to overwrite the hardware version: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/apis/v1beta1/types.go#L205

GitHub

cluster-api-provider-vsphere/apis/v1beta1/types.go at main · kubernetes-sigs/cluster-api-provider-vsphere

Contribute to kubernetes-sigs/cluster-api-provider-vsphere development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/apis/v1beta1/types.go#L205

Cristian Vlad

2025-03-11 14:37:50

@Cristian Vlad has joined the channel

Dragos Nicu

2025-03-11 15:02:10

@Dragos Nicu has joined the channel

jawnsy

2025-03-18 23:49:22

@jawnsy has joined the channel

Victor Sartori

2025-03-19 17:11:43

Hello everyone!

I came across a need that I couldn't find in Ansible for the Image Builder.
I need to set some kernel parameters for the OS via sysctl. These parameters must persist after a reboot (and should also prevent users from having to define them through init containers, for example).

What would be the most viable solution for this?
Should I create a new role that copies the parameters to /etc/sysctl.d/98-custom.conf?
Or would it be better to have a generic role that copies any local file to a specified path in the image? This way, we could also solve future issues related to custom files that need to be added to the image.

Depending on our discussion here, I can work on this task

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-19 17:33:33

My suggestion would be to use nodecustomroles_post and supply your own custom ansible role.

👍 Sriraman Srinivasan

Victor Sartori

2025-03-19 17:39:49

Got it Marcus!! Thanks for the tip!

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-19 17:40:56

If it turns out there are others interested in similar then we can port it upstream into image-builder but as of right now I haven’t seen anyone looking for similar that wasn’t very specific so might be best with the custom role. 🙂

Victor Sartori

2025-03-19 17:43:21

Very fair. I had a feeling that something to solve this kind of problem had already been considered.

Anand Kumar

2025-03-20 18:42:28

@Anand Kumar has joined the channel

Slackbot

2025-03-24 14:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-03-24 14:32:10

I'll be skipping today. I have a bunch of things I'd like to get ready for KubeCon next week 😅

👍 mboersma

Drew Hudson-Viles

2025-03-24 16:31:30

It looks like there isn't anything in the agenda anyway - shall we skip altogether?

mboersma

2025-03-24 18:20:58

Done

Jan

2025-03-24 19:22:50

@Jan has joined the channel

Abhay Krishna Arunachalam

2025-04-02 22:27:09

Hello Image-builder maintainers, when trying to build RHEL 9 OVAs in CI, we ran into this error for all Kubernetes versions

Build 'vsphere-iso.vsphere' errored after 3 minutes 29 seconds: No host is compatible with the virtual machine.

After some digging, I discovered this is because the RHEL 9 Guest OS type rhel9_64Guest was introduced in vSphere API release 7.0.1.0 (source), which is compatible with virtual machine hardware version 18 (source). But image-builder hardcodes the VM hardware version to 15, which doesn't support RHEL 9. We worked around this on our end by patching image-builder after which the builds succeeded. I'm upstreaming the patch in this PR. Kindly take a look and let me know if there are any concerns. Thank you!

GitHub

Update Packer VM hardware version for RHEL 9 OVA builds by abhay-krishna · Pull Request #1710 · kubernetes-sigs/image-builder

Change description This PR overrides the VM hardware version to 18 for RHEL 9 OVA builds, and adds RHEL 9 to the OS ID mapping in the OVA build script. The RHEL 9 Guest OS type was introduced in v...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1710

👍 mboersma

Abhay Krishna Arunachalam

2025-04-02 22:28:58

I'm also curious if others ran into this, I'd be surprised if they didn't thinking

Abhay Krishna Arunachalam

2025-04-03 19:20:36

cc @mboersma @Drew Hudson-Viles @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-03 19:35:21

check-success
Sorry for the delay. Currently at KubeCon and busy busy busy 😆

Abhay Krishna Arunachalam

2025-04-03 19:56:32

Ah sorry for the untimely ping! Really appreciate you reviewing and merging the PR despite your busy schedule! ty

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-03 20:01:35

Nah don't worry about it 🙂 Always appreciate the contributions!

🙌 Abhay Krishna Arunachalam

Raj Surve

2025-04-04 02:03:13

@Raj Surve has joined the channel

oivindoh

2025-04-04 11:19:38

@oivindoh has joined the channel

Slackbot

2025-04-07 15:30:05

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2025-04-07 16:21:53

Nothing on the agenda, but I'm happy to have office hours if anyone has anything to talk about. Speak up in the next few minutes if so... :-)

mboersma

2025-04-07 16:31:53

Ok, let's skip today.

Please add to the agenda for next time if you have something to discuss or present, or ask your questions here in the Slack channel. 😄

Drew Hudson-Viles

2025-04-07 17:03:24

Sorry, I've been stuck in meetings and didn't even see the pop-ups for slack 🤦‍♂️

Drew Hudson-Viles

2025-04-07 17:03:45

Brains still recovering from 12.5k people over 5 days 🤣🤣

:kubernetes: mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-08 05:15:51

:face_palm: sorry, I was out yesterday for my friends birthday and totally forgot shot this.

Thomas Güttler

2025-04-11 18:04:08

@Thomas Güttler has left the channel

shameemshah

2025-04-18 10:54:04

@shameemshah has joined the channel

Slackbot

2025-04-21 15:30:02

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-21 15:32:24

Anyone else no longer have access to the agenda? Or is it just me for some reason?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-21 15:32:51

Also, if possible I’d rather we skip today but can make it if needed

Drew Hudson-Viles

2025-04-21 15:59:03

Yeah I'm being prompted to request access. But yes, bank holiday weekend in the UK, so won't be available myself.

mboersma

2025-04-21 16:04:11

I have the same problem--I also lost access to another SCL Google Doc at the same time.

mboersma

2025-04-21 16:05:06

But I have access to CAPI and CAPZ docs, so it's not a matter of me not having the right perms. I think we need the actual document owner to refresh its permissions so Kubernetes in general can see it.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-21 16:12:16

Do you know who that is? 😅

mboersma

2025-04-21 16:13:24

I don't--maybe @jsturtevant does? Or hopefully someone in sig-cluster-lifecycle leadership, I can ask Fabrizio.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-22 08:13:11

I had a tab still open with the agenda to I managed to grab a backup of it before reloading.
You can view it here:
It's currently set to comment-only as I don't want this to become the new agenda as it's tied to Giant Swarm but I wanted to make sure we didn't lose the history 🙂

👍 Drew Hudson-Viles, mboersma

mboersma

2025-04-23 16:11:14

I think Fabrizio is an owner, he said he could help us move it to the appropriate k8s document area where apparently it should have been to avoid this problem. Crossing fingers.

👍 Drew Hudson-Viles

mboersma

2025-04-23 16:58:33

@Marcus Noble we couldn't find a new owner but Fabrizio created a new doc here:

Drew Hudson-Viles

2025-04-23 17:18:13

But... how will I know unless I can see the doc 😄

image.png

Ahmet Beyazoğlu

2025-04-22 09:13:20

@Ahmet Beyazoğlu has joined the channel

Ahmet Beyazoğlu

2025-04-22 10:45:35

Hello everyone,
We’ve encountered an issue while using kubernetes-sigs/image-builder to build Red Hat-based node images for Cluster API (CAPI) workload clusters.
After provisioning a workload cluster with these images, we’ve noticed that /etc/resolv.conf on the nodes includes two unexpected nameserver entries . These are not defined in our bootstrap data, cloud-init config, or Image Builder templates.
Example output of resolve.conf:

; Created by cloud-init automatically, do not edit.
Generated by NetworkManager
search foo.bar local
nameserver 10.x.x.1
nameserver 10.x.x.2

These additional nameservers are not configured by the user and their origin is unknown.

Has anyone seen similar behavior or could point us to where these values might be coming from during the build/provisioning process? (I think these nameservers come from DHCP scope during CI build, but they should not be in the template.)

Buid : build-node-ova-vsphere-rhel-8 (including with "packer-node.json" )
Rhel ISO: rhel-8.10-x86_64
Image builder version : cluster-node-image-builder-amd64:v0.1.41

Happy to share more details if needed. Thanks in advance!

Tomas Dabašinskas

2025-04-25 06:40:00

@Tomas Dabašinskas has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2025-04-29 11:32:28

Image-builder v0.1.42 is now available:

Thanks to all contributors! 💙

GitHub

Release v0.1.42 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.42">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.42</a> What's Changed Update docs for image-builder v0.1.41 by @mbo...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.42

🎉 Drew Hudson-Viles, mboersma

:kubernetes_intensifies: Drew Hudson-Viles, mboersma

Slackbot

2025-05-05 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles

2025-05-05 15:31:14

Another bank holiday in the UK and I'm away so I'm not going to be around for this I'm afraid. Sorry!

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-05 15:42:57

Same! ☝️

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-05 15:43:43

But it also seems like the agenda doc isn’t sorted yet(?) so no items anyway I guess

Arnaud Pons

2025-05-05 16:18:48

Hello all,

I've made a feature request regarding the support of ARM64 azure VM template build:

I've described the list of detected blockers based on what I've experienced on my side.
As said in the issue I'm not confortable to make a PR for this due to the lack of knowing on how you would like to see this feature handled.
Trying to add a single dedicated makefile target (i.e. sig ubuntu 22.04 ARM64) seems to cause a lot of duplicated code, and on the other hand, trying to add the notion of processor architecture more globaly needs to make a lot of modifications on things I don't know enough and I'm not able to test.

I'm open to feedback and will be happy to help where I can to see this feature supported 🙂.

Software Engineer

2025-05-15 16:38:34

@Software Engineer has joined the channel

Gokul Girish

2025-05-16 04:36:31

@Gokul Girish has joined the channel

Gokul Girish

2025-05-16 06:33:26

Hello Everyone,
I'm understanding that Ubuntu Latest Versions (22.04 and 24.04) do not supports preseeds anymore, so it's not referenced over here .

How can I configure the preseed - to add changes to cloudinit / auto-install file ? I tried updating base/preseed-efi.cfg.tmpl and 22.04.efi user-data.tmpl but I don't see it has reflected. Also in the logs on trying to build make build-raw-ubuntu-2204-efi I can see multiple preseeds and user-data (cloud inits) being logged. What's the order of precedence

Could someone please guide on this ?
TIA

GitHub

image-builder/images/capi/packer/raw/raw-ubuntu-2204-efi.json at main · kubernetes-sigs/image-builder

Tools for building Kubernetes disk images. Contribute to kubernetes-sigs/image-builder development by creating an account on GitHub.

Original URL: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/raw/raw-ubuntu-2204-efi.json

Sriraman Srinivasan

2025-05-16 14:11:32

@Marcus Noble / @mboersma Can you please take a look at PR. This adds ability for user to specify ansible roles to be run post the sysprep stage. Currently there are no hooks which allow user to run custom playbooks/roles after all the stages(including sysprep but before goss validation runs).
cc: @rajas

GitHub

Add ability to run playbooks post sysprep by sriramandev · Pull Request #1751 · kubernetes-sigs/image-builder

Change description There can be cases where we want to run or execute changes/collections post system prep stage(Towards the end but before goss validations). To enable this, added ability to run ...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1751

👍 Gokul Girish

Sriraman Srinivasan

2025-05-20 05:07:37

@Marcus Noble / @mboersma Can you please have a look at the above PR?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-20 05:09:24

Both me and Matt are currently away this week. Is it ok if we get to it next week when we’re back?

Sriraman Srinivasan

2025-05-20 05:21:14

@Marcus Noble Let me see if I can get someone else to look at these. Please do not bother your break. Sorry for pinging during your break(didn't realize that).

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-20 05:57:10

It’s totally fine. I’m actually away for work so I might get chance to look at it for you but it’s not guaranteed

🙏 Sriraman Srinivasan

Sriraman Srinivasan

2025-05-21 07:40:05

@chrischdi Can you please have a look at the PR ?

chrischdi

2025-05-21 18:52:56

(Did see this notification but not sure when I get back to it)

👍 Sriraman Srinivasan

Sriraman Srinivasan

2025-05-27 05:22:26

@Marcus Noble / @mboersma Once you are back, can you please have a look at the PR?

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-27 06:47:55

👍 I'm going to try and get to it today but I have a lot to catch up on so apologies if it takes me a while

👍 Sriraman Srinivasan

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-27 08:59:38

LGTM. Assigned to the others for approval but if no one comes back by this afternoon I'm happy to merge it 🙂

:ty_thankyou: Sriraman Srinivasan

mboersma

2025-05-27 16:17:18

Sorry, all, I'm back now. Thanks for being on top of things @Marcus Noble!

💙 Marcus Noble, Sriraman Srinivasan

mboersma

2025-05-16 15:51:35

Image-builder v0.1.43 is now available:

Thanks to all contributors! heart-kube

🎉 Sriraman Srinivasan

Slackbot

2025-05-19 15:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma

2025-05-19 16:20:11

We’ll have to skip today’s office hours, as the active maintainers are either on break or at an offsite.

Karine Santos

2025-05-20 21:38:34

@Karine Santos has joined the channel

Karine Santos

2025-05-21 00:39:11

[Packer Build Failing with "ssh: handshake failed: EOF" on Ubuntu 24.04]

Karine Santos

2025-05-21 00:42:14

Hi everyone,
I’m encountering an issue during a Packer build where the Ansible provisioner fails with an SSH handshake error. The build consistently crashes at the same step, and I’d appreciate any insights.
Environment

OS: Ubuntu 24.04 (LTS)

Packer version: 1.9.4 (QEMU builder)

Error:

==> qemu: ssh: handshake failed: EOF  
2025/05/20 23:11:46 [ERROR] ssh session open error: 'EOF', attempting reconnect

Current Configuration
I’ve already adjusted SSH settings for resilience:

{
  "builders": [
    {
      "accelerator": "{{user accelerator}}",
      "boot_command": [
        "{{user boot&#95;command&#95;prefix}}",
        "{{user boot&#95;media&#95;path}}",
        "{{user boot&#95;command&#95;suffix}}"
      ],
      "boot_wait": "{{user boot&#95;wait}}",
      "cd_files": [
        "{{user cd&#95;files}}"
      ],
      "cd_label": "cidata",
      "cpu_model": "host",
      "cpus": "{{user cpus}}",
      "disk_compression": "{{ user disk&#95;compression}}",
      "disk_discard": "{{user disk&#95;discard}}",
      "disk_image": "{{ user disk&#95;image }}",
      "disk_interface": "virtio-scsi",
      "disk_size": "{{user disk&#95;size}}",
      "firmware": "{{user firmware}}",
      "format": "{{user format}}",
      "headless": "{{user headless}}",
      "http_directory": "{{user http&#95;directory}}",
      "iso_checksum": "{{user iso&#95;checksum&#95;type}}:{{user iso&#95;checksum}}",
      "iso_url": "{{user iso&#95;url}}",
      "memory": "{{user memory}}",
      "net_device": "virtio-net",
      "output_directory": "{{user output&#95;directory}}",
      "qemu_binary": "{{user qemu&#95;binary}}",
      "shutdown_command": "echo '{{user ssh&#95;password}}' | sudo -S -E sh -c 'usermod -L {{user ssh&#95;username}} && {{user shutdown&#95;command}}'",
      "ssh_password": "{{user ssh&#95;password}}",
      "ssh_timeout": "2h",
      "ssh_username": "{{user ssh&#95;username}}",
      "type": "qemu",
      "vm_name": "{{user vm&#95;name}}",
      "vnc_bind_address": "{{user vnc&#95;bind&#95;address}}"
    }
  ],
  "post-processors": [
    {
      "environment_vars": [
        "CUSTOM_POST_PROCESSOR={{user custom&#95;post&#95;processor}}"
      ],
      "inline": [
        "if [ \"$CUSTOM_POST_PROCESSOR\" != \"true\" ]; then exit 0; fi",
        "{{user custom&#95;post&#95;processor&#95;command}}"
      ],
      "name": "custom-post-processor",
      "type": "shell-local"
    },
    {
      "environment_vars": [
        "OUTPUT_DIR={{user output&#95;directory}}",
        "ARTIFACT_NAME={{user artifact&#95;name}}",
        "KUBEVIRT={{user kubevirt}}"
      ],
      "inline": [
        "if [ \"$KUBEVIRT\" != \"true\" ]; then",
        "exit 0",
        "else",
        "bash ./packer/qemu/scripts/build_kubevirt_image.sh {{user build&#95;name}}-container-disk",
        "fi"
      ],
      "name": "kubevirt",
      "type": "shell-local"
    }
  ],
  "provisioners": [
    {
      "environment_vars": [
        "PYPY_HTTP_SOURCE={{user pypy&#95;http&#95;source}}"
      ],
      "execute_command": "BUILD_NAME={{user build&#95;name}}; if [[ \"${BUILD_NAME}\" == \"flatcar\" ]]; then sudo {{.Vars}} -S -E bash '{{.Path}}'; fi",
      "script": "./packer/files/flatcar/scripts/bootstrap-flatcar.sh",
      "type": "shell"
    },
    {
      "ansible_env_vars": [
        "ANSIBLE_SSH_ARGS='{{user existing&#95;ansible&#95;ssh&#95;args}} {{user ansible&#95;common&#95;ssh&#95;args}}'",
        "KUBEVIRT={{user kubevirt}}"
      ],
      "extra_arguments": [
        "--extra-vars",
        "{{user ansible&#95;common&#95;vars}}",
        "--extra-vars",
        "{{user ansible&#95;extra&#95;vars}}",
        "--extra-vars",
        "{{user ansible&#95;user&#95;vars}}",
        "--scp-extra-args",
        "{{user ansible&#95;scp&#95;extra&#95;args}}"
      ],
      "playbook_file": "./ansible/firstboot.yml",
      "type": "ansible",
      "user": "builder"
    },
    {
      "expect_disconnect": true,
      "inline": [
        "sudo reboot now"
      ],
      "inline_shebang": "/bin/bash -e",
      "type": "shell"
    },
    {
      "ansible_env_vars": [
        "ANSIBLE_SSH_ARGS='{{user existing&#95;ansible&#95;ssh&#95;args}} {{user ansible&#95;common&#95;ssh&#95;args}}'",
        "KUBEVIRT={{user kubevirt}}"
      ],
      "extra_arguments": [
        "--extra-vars",
        "{{user ansible&#95;common&#95;vars}}",
        "--extra-vars",
        "{{user ansible&#95;extra&#95;vars}}",
        "--extra-vars",
        "{{user ansible&#95;user&#95;vars}}",
        "--scp-extra-args",
        "{{user ansible&#95;scp&#95;extra&#95;args}}"
      ],
      "playbook_file": "./ansible/node.yml",
      "type": "ansible",
      "user": "builder"
    },
    {
      "arch": "{{user goss&#95;arch}}",
      "format": "{{user goss&#95;format}}",
      "format_options": "{{user goss&#95;format&#95;options}}",
      "goss_file": "{{user goss&#95;entry&#95;file}}",
      "inspect": "{{user goss&#95;inspect&#95;mode}}",
      "tests": [
        "{{user goss&#95;tests&#95;dir}}"
      ],
      "type": "goss",
      "url": "{{user goss&#95;url}}",
      "use_sudo": true,
      "vars_file": "{{user goss&#95;vars&#95;file}}",
      "vars_inline": {
        "ARCH": "amd64",
        "OS": "{{user distro&#95;name | lower}}",
        "OS_VERSION": "{{user distribution&#95;version | lower}}",
        "PROVIDER": "qemu",
        "containerd_version": "{{user containerd&#95;version}}",
        "kubernetes_cni_deb_version": "{{ user kubernetes&#95;cni&#95;deb&#95;version }}",
        "kubernetes_cni_rpm_version": "{{ split (user kubernetes&#95;cni&#95;rpm&#95;version) \"-\" 0 }}",
        "kubernetes_cni_source_type": "{{user kubernetes&#95;cni&#95;source&#95;type}}",
        "kubernetes_cni_version": "{{user kubernetes&#95;cni&#95;semver | replace \"v\" \"\" 1}}",
        "kubernetes_deb_version": "{{ user kubernetes&#95;deb&#95;version }}",
        "kubernetes_rpm_version": "{{ split (user kubernetes&#95;rpm&#95;version) \"-\" 0  }}",
        "kubernetes_source_type": "{{user kubernetes&#95;source&#95;type}}",
        "kubernetes_version": "{{user kubernetes&#95;semver | replace \"v\" \"\" 1}}"
      },
      "version": "{{user goss&#95;version}}"
    }
  ],
  "variables": {
    "accelerator": "kvm",
    "ansible_common_vars": "",
    "ansible_extra_vars": "ansible_python_interpreter=/usr/bin/python3",
    "ansible_user_vars": "",
    "artifact_name": "{{user build&#95;name}}-kube-{{user kubernetes&#95;semver}}",
    "boot_media_path": "http://{{ .HTTPIP }}:{{ .HTTPPort }}",
    "boot_wait": "10s",
    "build_timestamp": "{{timestamp}}",
    "cd_files": "linux/base/**.nothing",
    "containerd_sha256": null,
    "containerd_url": " containerd&#95;version}}/cri-containerd-cni-{{user containerd&#95;version}}-linux-amd64.tar.gz",
    "containerd_version": null,
    "cpus": "1",
    "crictl_url": " crictl&#95;version}}/crictl-v{{user crictl&#95;version}}-linux-amd64.tar.gz",
    "crictl_version": null,
    "disk_compression": "false",
    "disk_discard": "unmap",
    "disk_image": "false",
    "disk_size": "20480",
    "existing_ansible_ssh_args": "{{env ANSIBLE&#95;SSH&#95;ARGS}}",
    "firmware": "",
    "format": "qcow2",
    "headless": "true",
    "http_directory": "./packer/qemu/linux/{{user distro&#95;name}}/http/",
    "kubernetes_cni_deb_version": null,
    "kubernetes_cni_http_source": null,
    "kubernetes_cni_semver": null,
    "kubernetes_cni_source_type": null,
    "kubernetes_container_registry": null,
    "kubernetes_deb_gpg_key": null,
    "kubernetes_deb_repo": null,
    "kubernetes_deb_version": null,
    "kubernetes_http_source": null,
    "kubernetes_load_additional_imgs": null,
    "kubernetes_rpm_gpg_check": null,
    "kubernetes_rpm_gpg_key": null,
    "kubernetes_rpm_repo": null,
    "kubernetes_rpm_version": null,
    "kubernetes_semver": null,
    "kubernetes_series": null,
    "kubernetes_source_type": null,
    "machine_id_mode": "444",
    "memory": "2048",
    "oem_id": "",
    "output_directory": "./output/{{user build&#95;name}}-kube-{{user kubernetes&#95;semver}}",
    "python_path": "",
    "qemu_binary": "qemu-system-x86_64",
    "ssh_password": "$SSH_PASSWORD",
    "ssh_username": "builder",
    "vm_name": "{{user build&#95;name}}-kube-{{user kubernetes&#95;semver}}",
    "vnc_bind_address": "127.0.0.1"
  }
}

Debugging Steps Taken

Increased timeouts/retries: No change.

Verified credentials: Manual SSH works post-failure (if I pause the build).

Added verbose logging: Packer debug logs show the VM boots, but SSH dies abruptly.

Tested locally and in CI: Same error on both.

Suspected Causes

SSH service instability: The VM might not finish SSH setup before Packer connects.

Cloud-init/autoinstall misconfiguration: Ubuntu 24.04’s autoinstall could be skipping critical steps.

QEMU networking: NAT/port forwarding might interfere.

Questions for the Community

Has anyone hit this with Ubuntu 24.04’s autoinstall?

Are there known workarounds for Packer + QEMU SSH flakes?

How can I inspect the VM’s SSH service mid-build? (e.g., console logs?)

Additional Context

Packer boot command:

{
  "boot_command_prefix": "clinux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/24.04/'initrd /casper/initrdboot",
  "build_name": "ubuntu-2404",
  "distribution_version": "2404",
  "distro_name": "ubuntu",
  "guest_os_type": "ubuntu-64",
  "iso_checksum": "d6dab0c3a657988501b4bd76f1297c053df710e06e0c3aece60dead24f270b4d",
  "iso_checksum_type": "sha256",
  "iso_url": "",
  "os_display_name": "Ubuntu 24.04",
  "shutdown_command": "shutdown -P now",
  "unmount_iso": "true"
}

Any tips would be greatly appreciated! I’m happy to provide more logs or test suggestions.

Karine Santos

2025-05-21 17:03:07

/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/.local/bin/packer build -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/cni.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/containerd.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/wasm-shims.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/common.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/additional_components.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/ecr_credential_provider.json"  -color=true -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/qemu/qemu-ubuntu-2404.json"  packer/qemu/packer.json
8401qemu: output will be in this color.
8402==> qemu: Retrieving ISO
8403==> qemu: Trying 
8404==> qemu: Trying 
8405==> qemu: Download failed context deadline exceeded
8406==> qemu: error downloading ISO: [context deadline exceeded]
8407Build 'qemu' errored after 30 minutes 515 milliseconds: error downloading ISO: [context deadline exceeded]
8408==> Wait completed after 30 minutes 515 milliseconds
8409==> Some builds didn't complete successfully and had errors:
8410--> qemu: error downloading ISO: [context deadline exceeded]
8411==> Builds finished but no artifacts were created.
8412make[2]: ****** [Makefile:560: build-qemu-ubuntu-2404] Error 1
8413make[2]: Leaving directory '/builds/magalu-cloud-iaas/k8s/image-builder/images/capi'
8414make[1]: ****** [Makefile:1245: mgc-build-image] Error 2
8415make[1]: Leaving directory '/builds/magalu-cloud-iaas/k8s/image-builder/images/capi'
8416make: ****** [Makefile:37: mgc-build-image] Error 2

8417
Cleaning up project directory and file based variables
00:01
8418ERROR: Job failed: command terminated with exit code 1

Victor Sartori

2025-05-21 19:22:02

@Karine Santos Looking at the messages, this error seems related to the ISO download.
The first error indicates a timeout while downloading the ISO file.

👍 Sriraman Srinivasan

eszanon

2025-05-26 14:06:54

@eszanon has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2025-05-27 16:22:30

Image-builder v0.1.44 is now available:
Thanks to all contributors! 💙

GitHub

Release v0.1.44 · kubernetes-sigs/image-builder

This release of the image-builder container is available at: <a href="http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.44">registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.44</a> What's Changed Update docs for v0.1.43 by @mboersma in #1749...

Original URL: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.44

🎺 mboersma

:cluster_api: mboersma

:kubernetes_party: mboersma, Drew Hudson-Viles

Michael Nielson

2025-05-30 22:26:17

@Michael Nielson has joined the channel

Slackbot

2025-06-02 15:30:15

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-02 15:32:33

☝️ This is the updated agenda doc -
(@mboersma when you have time would you mind recreating that reminder with the new link?)

👍 mboersma

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-02 15:38:56

set the channel topic: Slack channel for the image-builder project: https://github.com/kubernetes-sigs/image-builder

Office Hours: https://docs.google.com/document/d/100uv2GmlgWyLBVP65W6ABNJ_EqbvVYTYtTilCLbnVYI/edit

mboersma

2025-06-02 16:04:06

set up a reminder “https://docs.google.com/document/d/100uv2GmlgWyLBVP65W6ABNJ_EqbvVYTYtTilCLbnVYI/edit” in this channel at 8:30AM every other Monday (next occurrence is June 16th), Mountain Daylight Time.

:thank_you_icon: Marcus Noble

mboersma

2025-06-02 16:05:46

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/100uv2GmlgWyLBVP65W6ABNJ_EqbvVYTYtTilCLbnVYI/edit” in this channel at 8:30AM every other Monday (next occurrence is June 16th), Mountain Daylight Time.

runzhliu

2025-06-04 16:26:58

@runzhliu has joined the channel

Sriraman Srinivasan

2025-06-09 08:17:08

@Marcus Noble / @mboersma PR needs your approvals for test and merger. The changeset primarily targets vSphere OVA builds.
cc: @palnabarun

GitHub

ova: make the root partition xfs for Photon 5, Ubuntu 22.04, and Ubuntu 24.04 by palnabarun · Pull Request #1766 · kubernetes-sigs/image-builder

Signed-off-by: Nabarun Pal <a href="mailto:pal.nabarun95@gmail.com">pal.nabarun95@gmail.com</a> Change description This PR updates the root filesystem to xfs from ext4 Related issues None Additional context None

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1766

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-09 08:25:22

It's on my list. Will try and get to it today but so far my day isn't going too well 😅

🙏 Sriraman Srinivasan

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-09 15:35:48

Is there any more context to this? There's no related issues nor any explanation why it's needed or what the impact would be.

Sriraman Srinivasan

2025-06-10 04:39:58

Will create a feature request with the details and also update the PR with the context. In nutshell, this more wrt to improving the node performance.

palnabarun

2025-06-11 11:16:46

@Marcus Noble added some context to . Thanks again for taking a look!

🙏 Sriraman Srinivasan

Sriraman Srinivasan

2025-06-11 13:03:30

@Marcus Noble Please do let me know if you need any further information.

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-12 08:41:55

👍 lgtm

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-12 08:42:03

Thank you for the context! 💙

chrischdi

2025-06-10 09:52:21

Hey folks, can I get a review for ? This re-adds the presubmit vsphere/OVA CI job (as optional) again, based on the new community based infra.

I’m doing the testing and required changes to make it finally work in

GitHub

[WIP] capi: adjust ova script for community infrastructure and fixes by chrischdi · Pull Request #1767 · kubernetes-sigs/image-builder

Change description Adjusts changes to make the pull-ova-all job work again xref: kubernetes/test-infra#34959 Related issues Fixes # Additional context

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1767

💙 Marcus Noble, rajas

:git_merge: dims

chrischdi

2025-06-11 09:49:05

Note: the image-builder PR would now be open for a review 🙂 the presubmit is green on that PR

Yuanliang Zhang

2025-06-10 18:13:27

@Yuanliang Zhang has joined the channel

Yuanliang Zhang

2025-06-10 18:15:43

Anybody aware of any known PR check-in failure /test pull-gcp-all? This is not related with my change Make kubelet starting as a windows service by zylxjtu · Pull Request #1752 · kubernetes-sigs/image-builder, so I'm wondering if there are any known (infra-related maybe?) issue? Thanks!

GitHub

Make kubelet starting as a windows service by zylxjtu · Pull Request #1752 · kubernetes-sigs/image-builder

Change description To support features such as windows graceful shutdown, we will need to make kubelet (itself) running as a windows service. this means we will have to remove the dependency on ns...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1752

palnabarun

2025-06-11 11:16:15

@palnabarun has joined the channel

Rémy Léone

2025-06-13 17:23:18

@Rémy Léone has joined the channel

Rémy Léone

2025-06-13 17:25:21

@Marcus Noble Hello 👋 I'm working with @Tomy Guichard at Scaleway with @Leïla MARABESE We can have a conversation about what you need for integrating image-builder with Scaleway 🙂

:gr_blob_wave: Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 07:21:37

👋 Hey y'all!

I just read the comment on the PR 🙂 Really happy to see Scaleway is willing to at least contribute effort to supporting this new provider! 💙 From my perspective having y'all down as reviewers for this provider would be enough (for now) for me to be happy to add this provider to image-builder. (@mboersma @Drew Hudson-Viles @jsturtevant do y'all agree? 🙂)

If possible I would LOVE to see some infra support from Scaleway to support testing of PR as we're very lacking in that regard across image-builder. It would go a long way to making Scaleway a solid, reliable option in image-builder but I also know how difficult these kind of agreements can be so I'm staying realistic 🙂

On a side note - very happy to see the Scaleway provider officially supported 😄 I'm a Scaleway user myself although just a single cluster plus some other resources.

GitHub

Add Scaleway packer provider by Tomy2e · Pull Request #1771 · kubernetes-sigs/image-builder

Change description This PR adds the Scaleway Packer Provider to the repo to allow users to provision CAPI images on Scaleway.

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1771#issuecomment-2970969011

🎉 Sriraman Srinivasan, rajas

Sriraman Srinivasan

2025-06-16 08:14:42

Welcome!!! 🎉

Drew Hudson-Viles

2025-06-16 08:21:15

Yo! Just here to echo Marcus really. Anything that can be provided be it in terms of supporting the provider and, if possible of course, infra for testing the provider would be amazing. Welcome!

:parrotk8s: Marcus Noble

Rémy Léone

2025-06-16 09:35:02

On a credentials sharing perspectives, what are the process on the CNCF side to share those kind of access? Are they organic and scope based, or is there within the CNCF a way for vendor to share this kind of resources ?

Rémy Léone

2025-06-16 09:35:27

Scaleway organization supports projects and we can have several projects within an organization

Rémy Léone

2025-06-16 09:35:33

We have support for IAM permissions

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 09:36:19

I would need to check with CNCF. I'm not totally sure myself. Maybe @mboersma knows more but he's currently on vacation.

Rémy Léone

2025-06-16 09:36:20

So I'm trying to envision what would be the best architecture for it, what would be the email address to invite in the organization, how to track credentials and have audit on who can access what

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 09:37:22

Maybe @bentheelder could share some insight on how providers go about donating cloud resources for testing 🙏

Sriraman Srinivasan

2025-06-16 09:39:06

@chrischdi may be also be able to provide some insight here?

chrischdi

2025-06-16 09:46:11

From my experience it is not only about sharing credentials. The infrastructure must be owned by the CNCF/Community if I know it right. So its more about donating credits or money so the community could leverage that. But I’m not the one who went through the process. Maybe best to ask in test-infra folks on what viable variants are.

chrischdi

2025-06-16 09:47:19

But in our case (vSphere) it was different, because we still use a public cloud (in our case Google Cloud) for the infrastructure, and are not a cloud provider ourselves.

bentheelder

2025-06-16 17:31:56

Please see this thread:

michael mccune (https://kubernetes.slack.com/team/U11HJAX1S)

hello infra folks, do we have some documentation about the cloud credits program where infrastructure providers can give credits to the kubernetes maintainers for the purposes of running tests on the provider's infrastructure?

Original URL: https://kubernetes.slack.com/archives/CCK68P2Q2/p1725993508886339

💙 Marcus Noble, Sriraman Srinivasan

:thank_you_icon: Marcus Noble

👍 Sriraman Srinivasan

Tomy Guichard

2025-06-13 17:25:25

@Tomy Guichard has joined the channel

Leïla MARABESE

2025-06-13 17:25:25

@Leïla MARABESE has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 10:25:42

⚠️ Looks like this Slack workspace if moving to the free tier this week 😳 Which if I understand correctly means we'll lose the history older than 90 days in this channel.

If there is anything from the history in this channel that you refer to please mention it in this thread and I'll try and get it backed up somewhere more permanent (GitHub or similar).

Changes to Kubernetes Slack | Kubernetes Contributors

Kubernetes Contributors

Changes to Kubernetes Slack

Kubernetes Slack will lose its special status and will be changing into a standard free Slack on June 20. Sometime later this year, our community will likely move to a new platform. If you are responsible for a channel or private channel, or a member of a User Group, you will need to take some actions as soon as you can. For the last decade, Slack has supported our project with a free customized enterprise account.

Original URL: https://www.kubernetes.dev/blog/2025/06/16/changes-to-kubernetes-slack-2025/#Kubernetes

👀 Drew Hudson-Viles

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 10:40:36

We're seeing some GCP failures in our PR tests, does anyone know / can confirm for me if the official Ubuntu 20.04 base image is no longer available on GCP now that support has ended?

googlecompute.ubuntu-2004: Error getting source image for instance creation: Could not find image, ubuntu-2004-lts, in projects

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 11:03:33

Unless anyone shouts at me not to, I propose removing Ubuntu 20.04 from GCP as it's EOL anyway.

GitHub

Remove Ubuntu 20.04 from GCP provider by AverageMarcus · Pull Request #1774 · kubernetes-sigs/image-builder

Change description Ubuntu 20.04 is now EOL and seems to have been removed from GCP so no longer works from image-builder. Related issues N/A Additional context /assign @drew-viles @mboersma @jstu...

Original URL: https://github.com/kubernetes-sigs/image-builder/pull/1774

👍 mboersma

Slackbot

2025-06-16 15:30:06

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 15:32:58

I’m not able to make it today but I have added some points to the agenda

mboersma

2025-06-16 15:55:34

I'll be there, hopefully others can join.

💙 Marcus Noble

:thx_thanks: Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 16:31:51

I might actually be able to going from my phone in about 5 min

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 17:09:04

Slight correction to what I said @mboersma - the public channels are archived but they’re offline and not currently searchable so we can’t reference them. (https://github.com/kubernetes/community/blob/master/communication/slack-migration-faq.md#what-information-will-we-lose)

GitHub

community/communication/slack-migration-faq.md at master · kubernetes/community

Kubernetes community content. Contribute to kubernetes/community development by creating an account on GitHub.

Original URL: https://github.com/kubernetes/community/blob/master/communication/slack-migration-faq.md#what-information-will-we-lose

Drew Hudson-Viles

2025-06-16 17:53:28

Sorry, my bad. Had to set off to pick Ada up from nursery early today and forgot to drop a message in here. Today has flown by so time got away with me and before I knew it I was heading out and already this is done 🤦‍♂️🤦‍♂️

Marcus Noble (k8s@marcusnoble.co.uk)

2025-06-16 17:56:24

No worries. Been a wild day for me too.

bentheelder

2025-06-16 17:31:46

@bentheelder has joined the channel

Public Channels

2 I already figured out, though It's quite tricky with packer to get all parameters right